Oct 30 18:35:56.707338 ip-10-0-131-224 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 18:35:57.089321 ip-10-0-131-224 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:35:57.089321 ip-10-0-131-224 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 18:35:57.089321 ip-10-0-131-224 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:35:57.089321 ip-10-0-131-224 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:35:57.097321 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097231    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:35:57.097321 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097318    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:35:57.097321 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097322    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:35:57.097321 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097326    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:35:57.097321 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097330    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:35:57.097512 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097333    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:35:57.097512 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097336    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:35:57.097512 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097339    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:35:57.097512 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097341    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:35:57.097512 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097344    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:35:57.097512 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097347    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:35:57.097512 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097349    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:35:57.097512 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097352    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:35:57.097512 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097355    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:35:57.097512 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097357    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:35:57.097512 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097360    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:35:57.097512 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097362    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:35:57.097512 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097365    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:35:57.097512 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097368    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:35:57.097512 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097371    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:35:57.097512 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097374    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:35:57.097512 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097377    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:35:57.097512 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097380    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:35:57.097512 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097383    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:35:57.097969 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097386    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:35:57.097969 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097388    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:35:57.097969 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097391    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:35:57.097969 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097394    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:35:57.097969 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097397    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:35:57.097969 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097399    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:35:57.097969 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097402    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:35:57.097969 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097404    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:35:57.097969 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097407    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:35:57.097969 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097410    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:35:57.097969 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097413    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:35:57.097969 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097415    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:35:57.097969 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097418    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:35:57.097969 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097421    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:35:57.097969 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097425    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:35:57.097969 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097430    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:35:57.097969 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097434    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:35:57.097969 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097437    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:35:57.097969 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097439    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:35:57.098426 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097442    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:35:57.098426 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097445    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:35:57.098426 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097448    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:35:57.098426 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097451    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:35:57.098426 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097453    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:35:57.098426 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097456    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:35:57.098426 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097458    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:35:57.098426 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097461    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:35:57.098426 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097464    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:35:57.098426 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097466    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:35:57.098426 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097469    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:35:57.098426 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097471    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:35:57.098426 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097474    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:35:57.098426 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097476    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:35:57.098426 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097479    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:35:57.098426 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097482    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:35:57.098426 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097484    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:35:57.098426 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097487    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:35:57.098426 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097490    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:35:57.098862 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097492    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:35:57.098862 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097495    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:35:57.098862 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097497    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:35:57.098862 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097501    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:35:57.098862 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097504    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:35:57.098862 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097506    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:35:57.098862 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097509    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:35:57.098862 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097512    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:35:57.098862 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097515    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:35:57.098862 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097518    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:35:57.098862 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097521    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:35:57.098862 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097524    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:35:57.098862 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097526    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:35:57.098862 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097530    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:35:57.098862 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097532    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:35:57.098862 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097535    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:35:57.098862 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097537    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:35:57.098862 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097540    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:35:57.098862 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097543    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:35:57.098862 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097545    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:35:57.099357 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097548    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:35:57.099357 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097552    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:35:57.099357 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097555    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:35:57.099357 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097557    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:35:57.099357 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097560    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:35:57.099357 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097563    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:35:57.099357 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097565    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:35:57.099357 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097568    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:35:57.099357 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097571    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:35:57.099357 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097574    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:35:57.099357 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097576    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:35:57.099357 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097579    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:35:57.099357 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097581    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:35:57.099357 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097584    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:35:57.099357 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097586    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:35:57.099357 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097589    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:35:57.099357 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097591    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:35:57.099357 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097594    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:35:57.099357 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097597    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:35:57.099816 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097602    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:35:57.099816 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097606    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:35:57.099816 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097609    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:35:57.099816 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097613    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:35:57.099816 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.097615    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:35:57.099816 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098419    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:35:57.099816 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098427    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:35:57.099816 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098431    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:35:57.099816 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098434    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:35:57.099816 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098437    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:35:57.099816 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098440    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:35:57.099816 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098443    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:35:57.099816 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098445    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:35:57.099816 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098448    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:35:57.099816 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098450    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:35:57.099816 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098454    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:35:57.099816 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098457    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:35:57.099816 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098460    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:35:57.099816 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098462    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:35:57.099816 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098465    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:35:57.100350 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098468    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:35:57.100350 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098470    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:35:57.100350 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098473    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:35:57.100350 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098475    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:35:57.100350 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098478    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:35:57.100350 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098482    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:35:57.100350 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098486    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:35:57.100350 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098489    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:35:57.100350 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098492    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:35:57.100350 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098495    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:35:57.100350 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098498    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:35:57.100350 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098501    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:35:57.100350 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098505    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:35:57.100350 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098508    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:35:57.100350 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098511    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:35:57.100350 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098513    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:35:57.100350 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098516    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:35:57.100350 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098519    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:35:57.100350 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098521    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:35:57.100805 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098524    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:35:57.100805 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098526    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:35:57.100805 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098529    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:35:57.100805 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098532    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:35:57.100805 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098534    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:35:57.100805 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098537    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:35:57.100805 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098539    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:35:57.100805 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098542    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:35:57.100805 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098544    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:35:57.100805 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098549    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:35:57.100805 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098554    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:35:57.100805 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098557    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:35:57.100805 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098561    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:35:57.100805 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098564    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:35:57.100805 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098567    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:35:57.100805 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098570    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:35:57.100805 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098572    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:35:57.100805 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098575    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:35:57.100805 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098577    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:35:57.101272 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098580    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:35:57.101272 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098582    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:35:57.101272 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098585    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:35:57.101272 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098588    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:35:57.101272 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098590    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:35:57.101272 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098593    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:35:57.101272 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098596    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:35:57.101272 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098599    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:35:57.101272 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098602    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:35:57.101272 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098604    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:35:57.101272 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098607    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:35:57.101272 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098609    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:35:57.101272 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098612    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:35:57.101272 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098615    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:35:57.101272 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098617    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:35:57.101272 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098620    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:35:57.101272 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098623    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:35:57.101272 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098625    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:35:57.101272 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098628    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:35:57.101728 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098631    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:35:57.101728 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098633    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:35:57.101728 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098635    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:35:57.101728 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098638    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:35:57.101728 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098641    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:35:57.101728 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098643    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:35:57.101728 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098646    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:35:57.101728 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098648    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:35:57.101728 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098651    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:35:57.101728 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098653    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:35:57.101728 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098656    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:35:57.101728 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098658    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:35:57.101728 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098660    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:35:57.101728 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098664    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:35:57.101728 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098666    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:35:57.101728 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098669    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:35:57.101728 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098671    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:35:57.101728 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098674    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:35:57.101728 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098677    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:35:57.101728 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098679    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:35:57.102221 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098682    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:35:57.102221 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098685    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:35:57.102221 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098688    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:35:57.102221 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098690    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:35:57.102221 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098692    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:35:57.102221 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098695    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:35:57.102221 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098697    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:35:57.102221 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098700    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:35:57.102221 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098703    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:35:57.102221 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098705    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:35:57.102221 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098708    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:35:57.102221 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098710    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:35:57.102221 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098713    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:35:57.102221 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.098716    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:35:57.102221 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099225    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:35:57.102221 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099231    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:35:57.102221 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099235    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:35:57.102221 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099238    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:35:57.102221 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099241    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:35:57.102668 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099244    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:35:57.102668 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099247    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:35:57.102668 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099250    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:35:57.102668 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099253    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:35:57.102668 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099255    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:35:57.102668 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099258    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:35:57.102668 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099261    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:35:57.102668 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099264    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:35:57.102668 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099266    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:35:57.102668 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099269    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:35:57.102668 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099271    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:35:57.102668 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099274    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:35:57.102668 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099278    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:35:57.102668 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099283    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:35:57.102668 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099286    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:35:57.102668 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099289    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:35:57.102668 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099292    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:35:57.102668 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099294    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:35:57.102668 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099297    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:35:57.103124 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099299    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:35:57.103124 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099302    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:35:57.103124 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099304    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:35:57.103124 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099307    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:35:57.103124 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099309    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:35:57.103124 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099312    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:35:57.103124 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099315    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:35:57.103124 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099317    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:35:57.103124 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099319    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:35:57.103124 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099322    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:35:57.103124 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099325    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:35:57.103124 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099327    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:35:57.103124 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099330    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:35:57.103124 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099333    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:35:57.103124 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099336    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:35:57.103124 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099339    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:35:57.103124 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099342    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:35:57.103124 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099344    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:35:57.103124 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099347    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:35:57.103124 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099349    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:35:57.103607 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099352    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:35:57.103607 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099354    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:35:57.103607 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099357    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:35:57.103607 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099360    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:35:57.103607 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099362    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:35:57.103607 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099365    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:35:57.103607 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099368    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:35:57.103607 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099370    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:35:57.103607 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099373    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:35:57.103607 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099376    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:35:57.103607 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099378    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:35:57.103607 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099380    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:35:57.103607 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099383    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:35:57.103607 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099385    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:35:57.103607 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099388    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:35:57.103607 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099390    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:35:57.103607 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099393    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:35:57.103607 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099395    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:35:57.103607 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099398    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:35:57.103607 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099400    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:35:57.104101 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099403    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:35:57.104101 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099406    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:35:57.104101 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099408    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:35:57.104101 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099411    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:35:57.104101 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099414    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:35:57.104101 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099434    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:35:57.104101 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099439    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:35:57.104101 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099445    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:35:57.104101 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099448    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:35:57.104101 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099451    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:35:57.104101 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099453    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:35:57.104101 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099456    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:35:57.104101 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099460    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:35:57.104101 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099462    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:35:57.104101 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099465    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:35:57.104101 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099467    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:35:57.104101 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099470    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:35:57.104101 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099473    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:35:57.104101 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099475    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:35:57.104101 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099478    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:35:57.104617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099480    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:35:57.104617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099483    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:35:57.104617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099486    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:35:57.104617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099489    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:35:57.104617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099492    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:35:57.104617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099496    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:35:57.104617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099499    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:35:57.104617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099502    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:35:57.104617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099505    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:35:57.104617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099507    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:35:57.104617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099518    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:35:57.104617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099521    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:35:57.104617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099524    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:35:57.104617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099526    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:35:57.104617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099529    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:35:57.104617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099532    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:35:57.104617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099535    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:35:57.104617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099539    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:35:57.104617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099542    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:35:57.105077 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099546    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:35:57.105077 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099548    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:35:57.105077 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.099551    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:35:57.105077 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099672    2572 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 18:35:57.105077 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099683    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 18:35:57.105077 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099692    2572 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 18:35:57.105077 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099698    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 18:35:57.105077 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099702    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 18:35:57.105077 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099705    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 18:35:57.105077 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099710    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 18:35:57.105077 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099714    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 18:35:57.105077 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099717    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 18:35:57.105077 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099720    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 18:35:57.105077 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099724    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 18:35:57.105077 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099727    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 18:35:57.105077 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099731    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 18:35:57.105077 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099734    2572 flags.go:64] FLAG: --cgroup-root=""
Oct 30 18:35:57.105077 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099737    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 18:35:57.105077 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099740    2572 flags.go:64] FLAG: --client-ca-file=""
Oct 30 18:35:57.105077 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099742    2572 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 18:35:57.105077 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099745    2572 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 18:35:57.105077 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099750    2572 flags.go:64] FLAG: --cluster-domain=""
Oct 30 18:35:57.105077 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099753    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 18:35:57.105616 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099758    2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 18:35:57.105616 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099761    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 18:35:57.105616 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099765    2572 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 18:35:57.105616 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099769    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 18:35:57.105616 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099772    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 18:35:57.105616 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099775    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 18:35:57.105616 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099778    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 18:35:57.105616 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099781    2572 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 18:35:57.105616 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099785    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 18:35:57.105616 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099790    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 18:35:57.105616 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099793    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 18:35:57.105616 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099796    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 18:35:57.105616 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099800    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 18:35:57.105616 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099803    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 18:35:57.105616 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099806    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 18:35:57.105616 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099809    2572 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 18:35:57.105616 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099812    2572 flags.go:64] FLAG: --enable-server="true"
Oct 30 18:35:57.105616 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099815    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 18:35:57.105616 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099820    2572 flags.go:64] FLAG: --event-burst="100"
Oct 30 18:35:57.105616 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099823    2572 flags.go:64] FLAG: --event-qps="50"
Oct 30 18:35:57.105616 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099826    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 18:35:57.105616 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099829    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 18:35:57.105616 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099832    2572 flags.go:64] FLAG: --eviction-hard=""
Oct 30 18:35:57.105616 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099837    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 18:35:57.106366 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099840    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 18:35:57.106366 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099843    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 18:35:57.106366 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099847    2572 flags.go:64] FLAG: --eviction-soft=""
Oct 30 18:35:57.106366 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099850    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 18:35:57.106366 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099853    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 18:35:57.106366 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099856    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 18:35:57.106366 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099859    2572 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 18:35:57.106366 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099862    2572 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 18:35:57.106366 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099866    2572 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 18:35:57.106366 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099869    2572 flags.go:64] FLAG: --feature-gates=""
Oct 30 18:35:57.106366 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099876    2572 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 18:35:57.106366 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099879    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 18:35:57.106366 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099882    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 18:35:57.106366 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099885    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 18:35:57.106366 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099888    2572 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 18:35:57.106366 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099892    2572 flags.go:64] FLAG: --help="false"
Oct 30 18:35:57.106366 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099895    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:35:57.106366 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099900    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 18:35:57.106366 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099903    2572 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 18:35:57.106366 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099906    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 18:35:57.106366 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099909    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 18:35:57.106366 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099913    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 18:35:57.106366 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099916    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 18:35:57.106924 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099919    2572 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 18:35:57.106924 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099922    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 18:35:57.106924 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099925    2572 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 18:35:57.106924 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099928    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 18:35:57.106924 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099931    2572 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 18:35:57.106924 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099934    2572 flags.go:64] FLAG: --kube-reserved=""
Oct 30 18:35:57.106924 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099936    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 18:35:57.106924 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099939    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 18:35:57.106924 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099942    2572 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 18:35:57.106924 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099945    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 18:35:57.106924 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099948    2572 flags.go:64] FLAG: --lock-file=""
Oct 30 18:35:57.106924 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099951    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 18:35:57.106924 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099954    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 18:35:57.106924 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099957    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 18:35:57.106924 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099963    2572 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 18:35:57.106924 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099966    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 18:35:57.106924 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099969    2572 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 18:35:57.106924 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099973    2572 flags.go:64] FLAG: --logging-format="text"
Oct 30 18:35:57.106924 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099976    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 18:35:57.106924 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099979    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 18:35:57.106924 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099983    2572 flags.go:64] FLAG: --manifest-url=""
Oct 30 18:35:57.106924 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099985    2572 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 18:35:57.106924 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099990    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 18:35:57.106924 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099993    2572 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 18:35:57.106924 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.099997    2572 flags.go:64] FLAG: --max-pods="110"
Oct 30 18:35:57.107537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100000    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 18:35:57.107537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100006    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 18:35:57.107537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100012    2572 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 18:35:57.107537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100017    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 18:35:57.107537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100021    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 18:35:57.107537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100024    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 18:35:57.107537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100027    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 18:35:57.107537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100034    2572 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 18:35:57.107537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100037    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 18:35:57.107537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100040    2572 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 18:35:57.107537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100043    2572 flags.go:64] FLAG: --pod-cidr=""
Oct 30 18:35:57.107537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100046    2572 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 18:35:57.107537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100049    2572 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 18:35:57.107537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100052    2572 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 18:35:57.107537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100055    2572 flags.go:64] FLAG: --port="10250"
Oct 30 18:35:57.107537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100058    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 18:35:57.107537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100073    2572 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0d1e1663a488cdaa6"
Oct 30 18:35:57.107537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100079    2572 flags.go:64] FLAG: --qos-reserved=""
Oct 30 18:35:57.107537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100084    2572 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 18:35:57.107537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100088    2572 flags.go:64] FLAG: --register-node="true"
Oct 30 18:35:57.107537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100091    2572 flags.go:64] FLAG: --register-with-taints=""
Oct 30 18:35:57.107537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100095    2572 flags.go:64] FLAG: --registry-burst="10"
Oct 30 18:35:57.107537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100098    2572 flags.go:64] FLAG: --registry-qps="5"
Oct 30 18:35:57.107537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100101    2572 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 18:35:57.107537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100106    2572 flags.go:64] FLAG: --reserved-memory=""
Oct 30 18:35:57.107537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100110    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 18:35:57.108199 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100113    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 18:35:57.108199 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100116    2572 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 18:35:57.108199 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100119    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 18:35:57.108199 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100121    2572 flags.go:64] FLAG: --runonce="false"
Oct 30 18:35:57.108199 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100125    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 18:35:57.108199 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100128    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 18:35:57.108199 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100131    2572 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 18:35:57.108199 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100135    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 18:35:57.108199 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100138    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 18:35:57.108199 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100142    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 18:35:57.108199 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100145    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 18:35:57.108199 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100148    2572 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 18:35:57.108199 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100151    2572 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 18:35:57.108199 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100154    2572 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 18:35:57.108199 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100157    2572 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 18:35:57.108199 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100160    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 18:35:57.108199 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100163    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 18:35:57.108199 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100167    2572 flags.go:64] FLAG: --system-cgroups=""
Oct 30 18:35:57.108199 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100169    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 18:35:57.108199 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100175    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 18:35:57.108199 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100178    2572 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 18:35:57.108199 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100181    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 18:35:57.108199 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100185    2572 flags.go:64] FLAG: --tls-min-version=""
Oct 30 18:35:57.108199 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100188    2572 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 18:35:57.108199 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100191    2572 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 18:35:57.108798 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100194    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 18:35:57.108798 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100197    2572 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 18:35:57.108798 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100200    2572 flags.go:64] FLAG: --v="2"
Oct 30 18:35:57.108798 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100205    2572 flags.go:64] FLAG: --version="false"
Oct 30 18:35:57.108798 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100209    2572 flags.go:64] FLAG: --vmodule=""
Oct 30 18:35:57.108798 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100214    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 18:35:57.108798 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100218    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 18:35:57.108798 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100354    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:35:57.108798 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100358    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:35:57.108798 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100361    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:35:57.108798 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100364    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:35:57.108798 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100367    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:35:57.108798 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100370    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:35:57.108798 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100372    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:35:57.108798 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100376    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:35:57.108798 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100379    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:35:57.108798 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100381    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:35:57.108798 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100384    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:35:57.108798 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100387    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:35:57.108798 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100390    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:35:57.108798 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100393    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:35:57.109315 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100396    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:35:57.109315 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100398    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:35:57.109315 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100401    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:35:57.109315 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100404    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:35:57.109315 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100406    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:35:57.109315 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100409    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:35:57.109315 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100412    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:35:57.109315 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100414    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:35:57.109315 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100417    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:35:57.109315 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100419    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:35:57.109315 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100422    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:35:57.109315 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100425    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:35:57.109315 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100427    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:35:57.109315 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100430    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:35:57.109315 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100433    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:35:57.109315 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100435    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:35:57.109315 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100439    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:35:57.109315 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100441    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:35:57.109315 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100444    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:35:57.109315 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100446    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:35:57.109786 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100449    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:35:57.109786 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100452    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:35:57.109786 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100454    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:35:57.109786 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100457    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:35:57.109786 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100459    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:35:57.109786 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100462    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:35:57.109786 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100465    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:35:57.109786 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100467    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:35:57.109786 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100470    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:35:57.109786 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100473    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:35:57.109786 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100475    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:35:57.109786 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100480    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:35:57.109786 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100484    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:35:57.109786 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100487    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:35:57.109786 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100490    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:35:57.109786 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100494    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:35:57.109786 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100497    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:35:57.109786 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100500    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:35:57.110233 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100503    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:35:57.110233 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100505    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:35:57.110233 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100508    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:35:57.110233 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100511    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:35:57.110233 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100513    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:35:57.110233 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100516    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:35:57.110233 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100518    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:35:57.110233 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100521    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:35:57.110233 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100524    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:35:57.110233 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100526    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:35:57.110233 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100529    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:35:57.110233 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100532    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:35:57.110233 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100534    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:35:57.110233 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100537    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:35:57.110233 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100539    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:35:57.110233 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100542    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:35:57.110233 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100544    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:35:57.110233 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100547    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:35:57.110233 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100549    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:35:57.110233 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100552    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:35:57.110705 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100555    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:35:57.110705 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100557    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:35:57.110705 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100560    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:35:57.110705 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100562    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:35:57.110705 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100565    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:35:57.110705 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100567    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:35:57.110705 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100570    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:35:57.110705 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100572    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:35:57.110705 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100575    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:35:57.110705 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100577    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:35:57.110705 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100580    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:35:57.110705 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100582    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:35:57.110705 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100585    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:35:57.110705 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100587    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:35:57.110705 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100590    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:35:57.110705 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100592    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:35:57.110705 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100595    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:35:57.110705 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100597    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:35:57.110705 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100600    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:35:57.110705 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100602    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:35:57.111186 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100605    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:35:57.111186 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100608    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:35:57.111186 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100610    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:35:57.111186 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100613    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:35:57.111186 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100616    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:35:57.111186 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100618    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:35:57.111186 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100621    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:35:57.111186 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100623    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:35:57.111186 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100626    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:35:57.111186 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100628    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:35:57.111186 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100631    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:35:57.111186 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100633    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:35:57.111186 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100636    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:35:57.111186 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100639    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:35:57.111186 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.100665    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:35:57.111186 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100702    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:35:57.111575 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100706    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:35:57.111575 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100710    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:35:57.111575 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100713    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:35:57.111575 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100716    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:35:57.111575 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100719    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:35:57.111575 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100721    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:35:57.111575 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100724    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:35:57.111575 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100726    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:35:57.111575 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100729    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:35:57.111575 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100732    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:35:57.111575 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100734    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:35:57.111575 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100738    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:35:57.111575 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100740    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:35:57.111575 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100743    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:35:57.111575 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100746    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:35:57.111575 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100748    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:35:57.111575 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100751    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:35:57.111575 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100753    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:35:57.111575 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100756    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:35:57.111575 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100759    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:35:57.112045 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100761    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:35:57.112045 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100764    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:35:57.112045 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100766    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:35:57.112045 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100769    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:35:57.112045 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100771    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:35:57.112045 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100774    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:35:57.112045 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100778    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:35:57.112045 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100781    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:35:57.112045 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100783    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:35:57.112045 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100786    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:35:57.112045 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100788    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:35:57.112045 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100791    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:35:57.112045 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100794    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:35:57.112045 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100796    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:35:57.112045 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100799    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:35:57.112045 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100802    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:35:57.112045 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100805    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:35:57.112045 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100807    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:35:57.112045 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100810    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:35:57.112045 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100813    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:35:57.112541 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100815    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:35:57.112541 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100818    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:35:57.112541 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100820    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:35:57.112541 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100823    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:35:57.112541 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100826    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:35:57.112541 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100828    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:35:57.112541 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100831    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:35:57.112541 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100833    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:35:57.112541 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100836    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:35:57.112541 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100838    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:35:57.112541 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100842    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:35:57.112541 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100845    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:35:57.112541 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100847    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:35:57.112541 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100850    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:35:57.112541 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100853    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:35:57.112541 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100855    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:35:57.112541 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100858    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:35:57.112541 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100861    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:35:57.112541 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100865    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:35:57.112541 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100868    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:35:57.113050 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100870    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:35:57.113050 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100873    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:35:57.113050 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100876    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:35:57.113050 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100878    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:35:57.113050 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100881    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:35:57.113050 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100883    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:35:57.113050 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100886    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:35:57.113050 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100889    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:35:57.113050 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100892    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:35:57.113050 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100894    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:35:57.113050 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100897    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:35:57.113050 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100900    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:35:57.113050 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100904    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:35:57.113050 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100908    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:35:57.113050 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100911    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:35:57.113050 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100914    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:35:57.113050 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100918    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:35:57.113050 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100921    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:35:57.113537 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100924    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:35:57.113537 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100926    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:35:57.113537 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100929    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:35:57.113537 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100931    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:35:57.113537 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100935    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:35:57.113537 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100938    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:35:57.113537 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100940    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:35:57.113537 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100943    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:35:57.113537 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100946    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:35:57.113537 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100949    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:35:57.113537 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100951    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:35:57.113537 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100954    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:35:57.113537 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100958    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:35:57.113537 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100960    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:35:57.113537 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100963    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:35:57.113537 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100965    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:35:57.113537 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100968    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:35:57.113537 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100971    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:35:57.113537 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100973    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:35:57.114038 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100976    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:35:57.114038 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100978    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:35:57.114038 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100981    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:35:57.114038 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100984    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:35:57.114038 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100988    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:35:57.114038 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100991    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:35:57.114038 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100993    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:35:57.114038 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.100996    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:35:57.114038 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.107565    2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 18:35:57.114038 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.107582    2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 18:35:57.114038 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.108675    2572 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 18:35:57.114038 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.108683    2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 18:35:57.114038 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108743    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:35:57.114038 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108749    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:35:57.114038 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108753    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:35:57.114038 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108756    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:35:57.114038 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108759    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:35:57.114038 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108762    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:35:57.114038 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108764    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:35:57.114518 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108767    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:35:57.114518 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108770    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:35:57.114518 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108772    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:35:57.114518 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108775    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:35:57.114518 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108778    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:35:57.114518 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108781    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:35:57.114518 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108784    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:35:57.114518 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108787    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:35:57.114518 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108789    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:35:57.114518 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108792    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:35:57.114518 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108794    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:35:57.114518 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108797    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:35:57.114518 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108800    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:35:57.114518 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108802    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:35:57.114518 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108805    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:35:57.114518 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108807    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:35:57.114518 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108810    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:35:57.114518 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108813    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:35:57.114518 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108816    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:35:57.114965 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108818    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:35:57.114965 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108821    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:35:57.114965 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108824    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:35:57.114965 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108834    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:35:57.114965 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108837    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:35:57.114965 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108839    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:35:57.114965 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108842    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:35:57.114965 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108845    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:35:57.114965 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108847    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:35:57.114965 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108850    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:35:57.114965 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108853    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:35:57.114965 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108855    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:35:57.114965 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108858    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:35:57.114965 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108860    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:35:57.114965 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108863    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:35:57.114965 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108865    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:35:57.114965 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108867    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:35:57.114965 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108872    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:35:57.114965 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108875    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:35:57.114965 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108878    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:35:57.115568 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108881    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:35:57.115568 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108884    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:35:57.115568 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108886    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:35:57.115568 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108889    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:35:57.115568 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108891    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:35:57.115568 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108894    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:35:57.115568 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108897    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:35:57.115568 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108899    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:35:57.115568 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108902    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:35:57.115568 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108905    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:35:57.115568 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108907    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:35:57.115568 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108910    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:35:57.115568 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108912    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:35:57.115568 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108915    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:35:57.115568 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108917    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:35:57.115568 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108921    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:35:57.115568 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108929    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:35:57.115568 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108932    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:35:57.115568 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108935    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:35:57.116016 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108937    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:35:57.116016 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108940    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:35:57.116016 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108942    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:35:57.116016 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108945    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:35:57.116016 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108948    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:35:57.116016 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108950    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:35:57.116016 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108953    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:35:57.116016 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108955    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:35:57.116016 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108958    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:35:57.116016 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108961    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:35:57.116016 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108963    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:35:57.116016 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108966    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:35:57.116016 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108968    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:35:57.116016 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108971    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:35:57.116016 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108973    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:35:57.116016 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108976    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:35:57.116016 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108978    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:35:57.116016 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108981    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:35:57.116016 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108983    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:35:57.116016 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108985    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:35:57.116507 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108988    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:35:57.116507 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108991    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:35:57.116507 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108994    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:35:57.116507 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108996    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:35:57.116507 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.108999    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:35:57.116507 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109001    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:35:57.116507 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109004    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:35:57.116507 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109006    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:35:57.116507 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109009    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:35:57.116507 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109011    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:35:57.116507 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109019    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:35:57.116507 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109022    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:35:57.116507 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109025    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:35:57.116507 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109029    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:35:57.116507 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109033    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:35:57.116507 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109036    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:35:57.116507 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109039    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:35:57.116507 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109041    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:35:57.116507 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109045    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:35:57.116507 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109047    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:35:57.117047 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109050    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:35:57.117047 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.109095    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:35:57.117047 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109277    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:35:57.117047 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109282    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:35:57.117047 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109285    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:35:57.117047 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109288    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:35:57.117047 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109291    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:35:57.117047 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109294    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:35:57.117047 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109296    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:35:57.117047 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109299    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:35:57.117047 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109302    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:35:57.117047 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109304    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:35:57.117047 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109307    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:35:57.117047 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109310    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:35:57.117047 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109313    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:35:57.117047 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109316    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:35:57.117565 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109319    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:35:57.117565 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109321    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:35:57.117565 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109324    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:35:57.117565 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109327    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:35:57.117565 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109329    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:35:57.117565 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109332    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:35:57.117565 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109335    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:35:57.117565 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109344    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:35:57.117565 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109347    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:35:57.117565 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109350    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:35:57.117565 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109352    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:35:57.117565 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109355    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:35:57.117565 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109358    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:35:57.117565 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109360    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:35:57.117565 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109362    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:35:57.117565 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109365    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:35:57.117565 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109368    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:35:57.117565 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109370    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:35:57.117565 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109373    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:35:57.117565 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109376    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:35:57.118182 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109378    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:35:57.118182 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109381    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:35:57.118182 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109383    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:35:57.118182 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109387    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:35:57.118182 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109391    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:35:57.118182 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109395    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:35:57.118182 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109397    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:35:57.118182 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109400    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:35:57.118182 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109403    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:35:57.118182 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109405    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:35:57.118182 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109407    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:35:57.118182 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109410    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:35:57.118182 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109412    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:35:57.118182 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109416    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:35:57.118182 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109420    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:35:57.118182 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109422    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:35:57.118182 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109425    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:35:57.118182 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109428    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:35:57.118617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109430    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:35:57.118617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109433    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:35:57.118617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109441    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:35:57.118617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109444    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:35:57.118617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109446    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:35:57.118617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109449    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:35:57.118617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109451    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:35:57.118617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109454    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:35:57.118617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109456    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:35:57.118617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109459    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:35:57.118617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109462    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:35:57.118617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109464    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:35:57.118617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109467    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:35:57.118617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109469    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:35:57.118617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109472    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:35:57.118617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109475    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:35:57.118617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109478    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:35:57.118617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109480    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:35:57.118617 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109482    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:35:57.119220 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109486    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:35:57.119220 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109488    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:35:57.119220 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109491    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:35:57.119220 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109499    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:35:57.119220 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109502    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:35:57.119220 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109505    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:35:57.119220 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109507    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:35:57.119220 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109510    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:35:57.119220 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109512    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:35:57.119220 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109515    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:35:57.119220 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109517    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:35:57.119220 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109520    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:35:57.119220 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109522    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:35:57.119220 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109525    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:35:57.119220 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109527    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:35:57.119220 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109530    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:35:57.119220 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109538    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:35:57.119220 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109540    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:35:57.119220 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109543    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:35:57.119220 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109545    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:35:57.119685 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109548    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:35:57.119685 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109550    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:35:57.119685 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109553    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:35:57.119685 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109555    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:35:57.119685 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109558    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:35:57.119685 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109560    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:35:57.119685 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109563    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:35:57.119685 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109565    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:35:57.119685 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109568    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:35:57.119685 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109570    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:35:57.119685 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109573    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:35:57.119685 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109575    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:35:57.119685 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109578    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:35:57.119685 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109581    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:35:57.119685 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109583    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:35:57.119685 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.109606    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:35:57.120078 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109643    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:35:57.120078 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109647    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:35:57.120078 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109649    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:35:57.120078 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109652    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:35:57.120078 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109654    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:35:57.120078 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109657    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:35:57.120078 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109659    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:35:57.120078 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109662    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:35:57.120078 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109664    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:35:57.120078 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109667    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:35:57.120078 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109669    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:35:57.120078 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109672    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:35:57.120078 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109674    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:35:57.120078 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109682    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:35:57.120078 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109685    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:35:57.120078 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109687    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:35:57.120078 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109690    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:35:57.120078 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109692    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:35:57.120078 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109695    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:35:57.120078 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109697    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:35:57.120579 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109700    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:35:57.120579 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109703    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:35:57.120579 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109712    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:35:57.120579 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109717    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:35:57.120579 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109720    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:35:57.120579 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109723    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:35:57.120579 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109726    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:35:57.120579 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109729    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:35:57.120579 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109732    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:35:57.120579 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109735    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:35:57.120579 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109737    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:35:57.120579 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109740    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:35:57.120579 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109744    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:35:57.120579 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109747    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:35:57.120579 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109750    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:35:57.120579 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109753    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:35:57.120579 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109756    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:35:57.120579 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109759    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:35:57.120579 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109761    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:35:57.121052 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109764    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:35:57.121052 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109766    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:35:57.121052 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109769    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:35:57.121052 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109771    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:35:57.121052 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109774    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:35:57.121052 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109777    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:35:57.121052 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109779    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:35:57.121052 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109788    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:35:57.121052 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109791    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:35:57.121052 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109794    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:35:57.121052 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109796    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:35:57.121052 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109799    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:35:57.121052 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109802    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:35:57.121052 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109805    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:35:57.121052 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109808    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:35:57.121052 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109810    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:35:57.121052 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109813    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:35:57.121052 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109816    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:35:57.121052 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109818    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:35:57.121506 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109821    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:35:57.121506 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109824    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:35:57.121506 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109826    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:35:57.121506 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109828    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:35:57.121506 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109831    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:35:57.121506 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109833    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:35:57.121506 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109836    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:35:57.121506 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109838    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:35:57.121506 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109841    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:35:57.121506 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109843    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:35:57.121506 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109846    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:35:57.121506 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109848    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:35:57.121506 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109850    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:35:57.121506 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109853    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:35:57.121506 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109855    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:35:57.121506 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109858    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:35:57.121506 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109860    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:35:57.121506 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109863    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:35:57.121506 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109865    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:35:57.121506 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109867    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:35:57.121966 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109870    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:35:57.121966 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109878    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:35:57.121966 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109880    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:35:57.121966 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109883    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:35:57.121966 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109885    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:35:57.121966 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109887    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:35:57.121966 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109890    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:35:57.121966 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109893    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:35:57.121966 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109895    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:35:57.121966 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109898    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:35:57.121966 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109900    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:35:57.121966 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109903    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:35:57.121966 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109905    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:35:57.121966 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109908    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:35:57.121966 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109910    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:35:57.121966 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109913    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:35:57.121966 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109915    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:35:57.121966 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109918    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:35:57.121966 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109920    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:35:57.122458 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109923    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:35:57.122458 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109926    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:35:57.122458 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109928    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:35:57.122458 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109931    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:35:57.122458 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109933    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:35:57.122458 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109936    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:35:57.122458 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109938    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:35:57.122458 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109940    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:35:57.122458 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:57.109943    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:35:57.122458 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.110005    2572 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 18:35:57.122458 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.112508    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 18:35:57.122458 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.113476    2572 server.go:1014] "Starting client certificate rotation"
Oct 30 18:35:57.122458 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.113576    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:35:57.122458 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.113619    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:35:57.122458 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.115559    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:35:57.122458 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.119096    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:35:57.129682 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.129667    2572 log.go:25] "Validated CRI v1 runtime API"
Oct 30 18:35:57.132561 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.132545    2572 log.go:25] "Validated CRI v1 image API"
Oct 30 18:35:57.133432 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.133407    2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 18:35:57.138696 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.138678    2572 fs.go:135] Filesystem UUIDs: map[73edc71b-6b7b-4a14-bbaa-705df54b3218:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e43d6a5c-0382-4583-8ce2-efaa9d60efbe:/dev/nvme0n1p4]
Oct 30 18:35:57.138747 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.138696    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 18:35:57.143825 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.143716    2572 manager.go:217] Machine: {Timestamp:2026-10-30 18:35:57.142353101 +0000 UTC m=+0.322375213 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3111194 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec270e2ddf6e3bb5b0636a821049e376 SystemUUID:ec270e2d-df6e-3bb5-b063-6a821049e376 BootID:1aa03d8d-b7ae-44f7-a873-6fb1856fd21e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:ff:55:2e:ab Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:ff:55:2e:ab Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:fb:b9:54:73:c2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 18:35:57.143825 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.143823    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 18:35:57.143951 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.143900    2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 18:35:57.147138 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.147113    2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 18:35:57.147268 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.147139    2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-224.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 18:35:57.147308 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.147271    2572 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 18:35:57.147308 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.147279    2572 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 18:35:57.147367 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.147325    2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:35:57.147367 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.147345    2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:35:57.148051 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.148040    2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 18:35:57.151956 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.151944    2572 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 18:35:57.152084 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.152075    2572 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 18:35:57.152675 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.152666    2572 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 18:35:57.152744 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.152678    2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 18:35:57.152744 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.152689    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 18:35:57.152744 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.152708    2572 kubelet.go:405] "Adding apiserver pod source"
Oct 30 18:35:57.152744 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.152724    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 18:35:57.153188 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.153174    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:35:57.155579 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.155552    2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 18:35:57.161263 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.161248    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:35:57.162010 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.161998    2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 18:35:57.162072 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.162028    2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 18:35:57.162202 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.162190    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 18:35:57.162236 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.162206    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 18:35:57.162236 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.162212    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 18:35:57.162236 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.162218    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 18:35:57.162236 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.162224    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 18:35:57.162236 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.162229    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 18:35:57.162913 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.162905    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 18:35:57.162946 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.162915    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 18:35:57.162946 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.162921    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 18:35:57.162946 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.162927    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 18:35:57.162946 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.162935    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 18:35:57.162946 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.162944    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 18:35:57.163967 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.163957    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 18:35:57.164000 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.163971    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 18:35:57.167592 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.167573    2572 server.go:1263] "Started kubelet"
Oct 30 18:35:57.167696 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.167669    2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 18:35:57.167753 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.167675    2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 18:35:57.167753 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.167731    2572 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 18:35:57.168296 ip-10-0-131-224 systemd[1]: Started Kubernetes Kubelet.
Oct 30 18:35:57.168803 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.168787    2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 18:35:57.170294 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.170279    2572 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 18:35:57.173636 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.173617    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 18:35:57.173742 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.173727    2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 18:35:57.175112 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.175088    2572 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 18:35:57.175112 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.175113    2572 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 18:35:57.175288 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.175211    2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 18:35:57.175288 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.175259    2572 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 18:35:57.175288 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.175265    2572 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 18:35:57.175468 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:57.175444    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-224.us-west-2.compute.internal\" not found"
Oct 30 18:35:57.176720 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.176705    2572 factory.go:55] Registering systemd factory
Oct 30 18:35:57.176720 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.176721    2572 factory.go:223] Registration of the systemd container factory successfully
Oct 30 18:35:57.176849 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:57.176753    2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 18:35:57.176952 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.176939    2572 factory.go:153] Registering CRI-O factory
Oct 30 18:35:57.177010 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.176956    2572 factory.go:223] Registration of the crio container factory successfully
Oct 30 18:35:57.177010 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.177006    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 18:35:57.177120 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.177023    2572 factory.go:103] Registering Raw factory
Oct 30 18:35:57.177120 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.177038    2572 manager.go:1199] Started watching for new ooms in manager
Oct 30 18:35:57.177885 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.177871    2572 manager.go:319] Starting recovery of all containers
Oct 30 18:35:57.186248 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.186213    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 18:35:57.188420 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.188402    2572 manager.go:324] Recovery completed
Oct 30 18:35:57.195846 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.195828    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:35:57.197705 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.197675    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:35:57.197778 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.197721    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:35:57.197778 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.197733    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:35:57.198324 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.198311    2572 cpu_manager.go:226] "Starting" policy="none"
Oct 30 18:35:57.198324 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.198319    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:35:57.198391 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.198323    2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 18:35:57.198391 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.198346    2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 18:35:57.199020 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.199006    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:35:57.199128 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.199033    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:35:57.199128 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.199044    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:35:57.200771 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.200755    2572 policy_none.go:50] "Start"
Oct 30 18:35:57.200854 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.200774    2572 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 18:35:57.200854 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.200788    2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 18:35:57.202879 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.202865    2572 policy_none.go:44] "Start"
Oct 30 18:35:57.237821 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.237795    2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 18:35:57.243880 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.243860    2572 manager.go:341] "Starting Device Plugin manager"
Oct 30 18:35:57.243961 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:57.243902    2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 18:35:57.243961 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.243914    2572 server.go:87] "Starting device plugin registration server"
Oct 30 18:35:57.244197 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.244184    2572 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 18:35:57.244276 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.244198    2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 18:35:57.244325 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.244299    2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 18:35:57.244395 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.244378    2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 18:35:57.244395 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.244390    2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 18:35:57.244972 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:57.244955    2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 18:35:57.245030 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:57.245007    2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-224.us-west-2.compute.internal\" not found"
Oct 30 18:35:57.301721 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:57.301693    2572 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-224.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 18:35:57.301721 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.301701    2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-224.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:35:57.303193 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:57.301686    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-224.us-west-2.compute.internal.18e3627b075bd828  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-224.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-224.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:35:57.167548456 +0000 UTC m=+0.347570569,LastTimestamp:2026-10-30 18:35:57.167548456 +0000 UTC m=+0.347570569,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-224.us-west-2.compute.internal,}"
Oct 30 18:35:57.306225 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.306203    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 18:35:57.306225 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.306224    2572 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 18:35:57.306352 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.306245    2572 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 18:35:57.306352 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:57.306282    2572 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 18:35:57.344837 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.344793    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:35:57.345576 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.345562    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:35:57.345641 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.345589    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:35:57.345641 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.345601    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:35:57.345641 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.345632    2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:35:57.359556 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:57.359529    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 18:35:57.359556 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:57.359529    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-224.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 18:35:57.359719 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:57.359549    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-224.us-west-2.compute.internal.18e3627b092800d2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-224.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-224.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-224.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:35:57.197705426 +0000 UTC m=+0.377727541,LastTimestamp:2026-10-30 18:35:57.197705426 +0000 UTC m=+0.377727541,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-224.us-west-2.compute.internal,}"
Oct 30 18:35:57.406874 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.406847    2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal"]
Oct 30 18:35:57.406970 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.406916    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:35:57.407724 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.407710    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:35:57.407795 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.407737    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:35:57.407795 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.407747    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:35:57.408323 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.408309    2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 18:35:57.408361 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.408346    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7zbwt"
Oct 30 18:35:57.409526 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.409513    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:35:57.409695 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.409681    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:35:57.409732 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.409713    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:35:57.410284 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.410259    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:35:57.410284 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.410285    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:35:57.410459 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.410299    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:35:57.410459 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.410318    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:35:57.410459 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.410364    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:35:57.410459 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.410382    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:35:57.412472 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.412458    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:35:57.412533 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.412485    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:35:57.413241 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.413225    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:35:57.413319 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.413257    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:35:57.413319 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.413271    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:35:57.434790 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:57.434642    2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-224.us-west-2.compute.internal\" not found" node="ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:35:57.437455 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:57.437440    2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-224.us-west-2.compute.internal\" not found" node="ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:35:57.448556 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:57.448537    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 18:35:57.448650 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:57.448582    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Oct 30 18:35:57.448650 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:57.448549    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-224.us-west-2.compute.internal.18e3627b092854e4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-224.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-224.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-224.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:35:57.197726948 +0000 UTC m=+0.377749060,LastTimestamp:2026-10-30 18:35:57.197726948 +0000 UTC m=+0.377749060,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-224.us-west-2.compute.internal,}"
Oct 30 18:35:57.476838 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.476815    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0bdd5feb35ab75fe20b4acd57c05f061-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal\" (UID: \"0bdd5feb35ab75fe20b4acd57c05f061\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:35:57.476919 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.476848    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0bdd5feb35ab75fe20b4acd57c05f061-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal\" (UID: \"0bdd5feb35ab75fe20b4acd57c05f061\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:35:57.476919 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.476879    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e58124d8b2f46cf7f8af1e433d1d9a6-config\") pod \"kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal\" (UID: \"2e58124d8b2f46cf7f8af1e433d1d9a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:35:57.509963 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:57.509941    2572 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-224.us-west-2.compute.internal\" not found" node="ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:35:57.577775 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.577753    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0bdd5feb35ab75fe20b4acd57c05f061-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal\" (UID: \"0bdd5feb35ab75fe20b4acd57c05f061\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:35:57.577775 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.577780    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0bdd5feb35ab75fe20b4acd57c05f061-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal\" (UID: \"0bdd5feb35ab75fe20b4acd57c05f061\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:35:57.577946 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.577803    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e58124d8b2f46cf7f8af1e433d1d9a6-config\") pod \"kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal\" (UID: \"2e58124d8b2f46cf7f8af1e433d1d9a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:35:57.577946 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.577832    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e58124d8b2f46cf7f8af1e433d1d9a6-config\") pod \"kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal\" (UID: \"2e58124d8b2f46cf7f8af1e433d1d9a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:35:57.577946 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.577857    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0bdd5feb35ab75fe20b4acd57c05f061-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal\" (UID: \"0bdd5feb35ab75fe20b4acd57c05f061\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:35:57.577946 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.577864    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0bdd5feb35ab75fe20b4acd57c05f061-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal\" (UID: \"0bdd5feb35ab75fe20b4acd57c05f061\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:35:57.603097 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:57.603036    2572 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-224.us-west-2.compute.internal\" not found" node="ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:35:57.711021 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.710979    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:35:57.712158 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.712133    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:35:57.712298 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.712168    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:35:57.712298 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.712183    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:35:57.712298 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.712219    2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:35:57.737703 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.737670    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:35:57.740815 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.740797    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:35:57.767296 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:57.767278    2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:35:57.767355 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:57.767301    2572 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-224.us-west-2.compute.internal\": node \"ip-10-0-131-224.us-west-2.compute.internal\" not found"
Oct 30 18:35:57.874817 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:57.874746    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-224.us-west-2.compute.internal\" not found"
Oct 30 18:35:57.975748 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:57.975720    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-224.us-west-2.compute.internal\" not found"
Oct 30 18:35:58.076699 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:58.076671    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-224.us-west-2.compute.internal\" not found"
Oct 30 18:35:58.113806 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:58.113781    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 18:35:58.114220 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:58.113947    2572 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a511c06f6c4c24b5eb9148195e6ce288-46029cde054419ce.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-131-224.us-west-2.compute.internal.18e3627b092854e4\": read tcp 10.0.131.224:37638->77.112.254.244:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-131-224.us-west-2.compute.internal.18e3627b092854e4  default   4777 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-224.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-224.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-224.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:35:57 +0000 UTC,LastTimestamp:2026-10-30 18:35:57.407741673 +0000 UTC m=+0.587763785,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-224.us-west-2.compute.internal,}"
Oct 30 18:35:58.177809 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:58.177741    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-224.us-west-2.compute.internal\" not found"
Oct 30 18:35:58.277850 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:58.277825    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-224.us-west-2.compute.internal\" not found"
Oct 30 18:35:58.351125 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:58.351100    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:35:58.351972 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:58.351956    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:35:58.352056 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:58.351986    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:35:58.352056 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:58.351996    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:35:58.378473 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:58.378455    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-224.us-west-2.compute.internal\" not found"
Oct 30 18:35:58.412728 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:58.412698    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:30:57 +0000 UTC" deadline="2028-06-09 13:49:03.069693987 +0000 UTC"
Oct 30 18:35:58.412728 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:58.412723    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14107h13m4.656974257s"
Oct 30 18:35:58.452644 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:58.452588    2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:35:58.479520 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:58.479498    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-224.us-west-2.compute.internal\" not found"
Oct 30 18:35:58.580014 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:58.579989    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-224.us-west-2.compute.internal\" not found"
Oct 30 18:35:58.627299 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:58.627278    2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:35:58.680099 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:58.680059    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-224.us-west-2.compute.internal\" not found"
Oct 30 18:35:58.781231 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:58.781158    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-224.us-west-2.compute.internal\" not found"
Oct 30 18:35:58.819670 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:58.819635    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bdd5feb35ab75fe20b4acd57c05f061.slice/crio-e4755db47b4602d74bc92410b4139a0c03ec31fe285b49cc74c99928d8351c06 WatchSource:0}: Error finding container e4755db47b4602d74bc92410b4139a0c03ec31fe285b49cc74c99928d8351c06: Status 404 returned error can't find the container with id e4755db47b4602d74bc92410b4139a0c03ec31fe285b49cc74c99928d8351c06
Oct 30 18:35:58.819956 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:58.819934    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e58124d8b2f46cf7f8af1e433d1d9a6.slice/crio-3c56149a6c66eaa82146cbbab520741d290fbb1283425fd10aa2409a3f28accf WatchSource:0}: Error finding container 3c56149a6c66eaa82146cbbab520741d290fbb1283425fd10aa2409a3f28accf: Status 404 returned error can't find the container with id 3c56149a6c66eaa82146cbbab520741d290fbb1283425fd10aa2409a3f28accf
Oct 30 18:35:58.827409 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:58.827382    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:35:58.847971 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:58.847925    2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:35:58.881504 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:58.881474    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-224.us-west-2.compute.internal\" not found"
Oct 30 18:35:58.965119 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:58.965103    2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:35:58.975659 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:58.975633    2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:35:59.039920 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.039861    2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:35:59.040975 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.040946    2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:35:59.108621 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.108601    2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:35:59.154747 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.154722    2572 apiserver.go:52] "Watching apiserver"
Oct 30 18:35:59.176497 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.176476    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 18:35:59.267871 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.267852    2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 18:35:59.268212 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.268193    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-h44bp","openshift-network-diagnostics/network-check-target-xcmps","openshift-network-operator/iptables-alerter-44cfg","kube-system/global-pull-secret-syncer-vs8lx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz","openshift-multus/multus-wd7km","openshift-multus/network-metrics-daemon-sqkfq","kube-system/konnectivity-agent-5v2r9","kube-system/kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal","openshift-dns/node-resolver-cdf94","openshift-image-registry/node-ca-vtp2j","openshift-ovn-kubernetes/ovnkube-node-mgd4r","openshift-cluster-node-tuning-operator/tuned-bhxxm"]
Oct 30 18:35:59.272241 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.272224    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.274583 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.274557    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.276856 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.276837    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:35:59.276951 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:59.276907    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xcmps" podUID="0b0aa783-d4ab-490c-a5f7-d7da5ad8d278"
Oct 30 18:35:59.279429 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.279200    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-44cfg"
Oct 30 18:35:59.281554 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.281534    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:35:59.281652 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:59.281599    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8lx" podUID="92f38b64-26fb-42d6-826a-94cd37e8b668"
Oct 30 18:35:59.284314 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.284291    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:35:59.286655 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.286639    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5v2r9"
Oct 30 18:35:59.287323 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.287300    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-run-ovn\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.287400 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.287344    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-cni-netd\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.287400 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.287367    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-multus-conf-dir\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.287400 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.287385    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tsxg\" (UniqueName: \"kubernetes.io/projected/e1c10266-f60d-4b43-b2ee-cef69669bc7b-kube-api-access-4tsxg\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.287536 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.287447    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-multus-cni-dir\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.287604 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.287584    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74a85732-6555-48bf-9ece-4e863cd1fb93-cni-binary-copy\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.287662 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.287623    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-run-netns\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.287662 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.287650    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-var-lib-kubelet\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.287741 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.287676    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-run-multus-certs\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.287741 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.287705    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-run-systemd\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.287803 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.287738    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-run-ovn-kubernetes\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.287803 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.287788    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.287908 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.287826    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1c10266-f60d-4b43-b2ee-cef69669bc7b-ovnkube-config\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.287908 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.287853    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1c10266-f60d-4b43-b2ee-cef69669bc7b-ovn-node-metrics-cert\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.287985 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.287919    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-var-lib-cni-bin\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.287985 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.287938    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-systemd-units\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.287985 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.287953    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-cni-bin\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.288200 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.287990    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-system-cni-dir\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.288200 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.288030    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-multus-socket-dir-parent\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.288200 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.288058    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-var-lib-openvswitch\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.288200 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.288102    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-run-openvswitch\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.288200 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.288126    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-log-socket\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.288200 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.288166    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb856\" (UniqueName: \"kubernetes.io/projected/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278-kube-api-access-vb856\") pod \"network-check-target-xcmps\" (UID: \"0b0aa783-d4ab-490c-a5f7-d7da5ad8d278\") " pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:35:59.288419 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.288213    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-run-k8s-cni-cncf-io\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.288419 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.288239    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-var-lib-cni-multus\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.288419 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.288285    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-node-log\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.288419 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.288317    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-kubelet\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.288419 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.288356    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-slash\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.288419 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.288371    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-run-netns\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.288419 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.288397    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-etc-openvswitch\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.288741 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.288469    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1c10266-f60d-4b43-b2ee-cef69669bc7b-env-overrides\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.288741 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.288501    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1c10266-f60d-4b43-b2ee-cef69669bc7b-ovnkube-script-lib\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.288741 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.288526    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-cnibin\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.288741 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.288563    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-hostroot\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.288741 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.288600    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/74a85732-6555-48bf-9ece-4e863cd1fb93-multus-daemon-config\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.288741 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.288629    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-etc-kubernetes\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.288741 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.288654    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkl9j\" (UniqueName: \"kubernetes.io/projected/74a85732-6555-48bf-9ece-4e863cd1fb93-kube-api-access-qkl9j\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.289056 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.289040    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.291510 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.291495    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:35:59.293895 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.293870    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:35:59.293971 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:59.293955    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqkfq" podUID="6df0f1e7-07b7-4231-bc70-ba81aef29856"
Oct 30 18:35:59.296519 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.296503    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cdf94"
Oct 30 18:35:59.298831 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.298814    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vtp2j"
Oct 30 18:35:59.310683 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.310644    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal" event={"ID":"2e58124d8b2f46cf7f8af1e433d1d9a6","Type":"ContainerStarted","Data":"3c56149a6c66eaa82146cbbab520741d290fbb1283425fd10aa2409a3f28accf"}
Oct 30 18:35:59.313687 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.313664    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal" event={"ID":"0bdd5feb35ab75fe20b4acd57c05f061","Type":"ContainerStarted","Data":"e4755db47b4602d74bc92410b4139a0c03ec31fe285b49cc74c99928d8351c06"}
Oct 30 18:35:59.330659 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.330641    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5mz5g\""
Oct 30 18:35:59.331459 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.331437    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 18:35:59.331562 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.331487    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 18:35:59.331804 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.331788    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 18:35:59.331872 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.331809    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 18:35:59.331964 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.331950    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 18:35:59.332200 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.332185    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tlfn7\""
Oct 30 18:35:59.332974 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.332960    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 18:35:59.333130 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.333117    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 18:35:59.333329 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.333316    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 18:35:59.333435 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.333424    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 18:35:59.333510 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.333497    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 18:35:59.336850 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.336835    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dnpxx\""
Oct 30 18:35:59.378215 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.378194    2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 18:35:59.389419 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389400    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9849d01-613f-439d-86de-b517ee4a3341-host\") pod \"node-ca-vtp2j\" (UID: \"b9849d01-613f-439d-86de-b517ee4a3341\") " pod="openshift-image-registry/node-ca-vtp2j"
Oct 30 18:35:59.389497 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389428    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-var-lib-openvswitch\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.389497 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389445    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-run-openvswitch\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.389497 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389460    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-var-lib-cni-multus\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.389497 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389491    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-run-openvswitch\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.389666 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389502    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-var-lib-openvswitch\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.389666 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389519    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-systemd\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.389666 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389549    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-run\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.389666 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389555    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-var-lib-cni-multus\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.389666 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389568    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhnjp\" (UniqueName: \"kubernetes.io/projected/90df09a3-8706-4883-b711-be63e8e9f191-kube-api-access-hhnjp\") pod \"node-resolver-cdf94\" (UID: \"90df09a3-8706-4883-b711-be63e8e9f191\") " pod="openshift-dns/node-resolver-cdf94"
Oct 30 18:35:59.389666 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389610    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92f38b64-26fb-42d6-826a-94cd37e8b668-kubelet-config\") pod \"global-pull-secret-syncer-vs8lx\" (UID: \"92f38b64-26fb-42d6-826a-94cd37e8b668\") " pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:35:59.389666 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389632    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-node-log\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.389666 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389648    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-kubelet\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.389666 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389666    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-slash\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.390039 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389685    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-run-netns\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.390039 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389700    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-etc-openvswitch\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.390039 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389717    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-kubelet\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.390039 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389716    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-node-log\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.390039 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389729    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1c10266-f60d-4b43-b2ee-cef69669bc7b-ovnkube-script-lib\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.390039 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389746    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-slash\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.390039 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389757    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-cnibin\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.390039 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389769    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-run-netns\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.390039 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389781    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-etc-openvswitch\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.390039 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389795    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-cnibin\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.390039 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389793    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-hostroot\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.390039 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389822    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-hostroot\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.390039 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389863    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-host\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.390039 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389892    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6bwn\" (UniqueName: \"kubernetes.io/projected/3935b0fd-69be-4272-adc5-ecfbc7af04ae-kube-api-access-g6bwn\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.390039 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389920    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p6bb\" (UniqueName: \"kubernetes.io/projected/9092fa02-cc22-41a2-b4ee-070298ecab99-kube-api-access-4p6bb\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:35:59.390039 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389948    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65beb5a5-1df3-42c8-9e14-769b4ad47255-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:35:59.390039 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389973    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90df09a3-8706-4883-b711-be63e8e9f191-tmp-dir\") pod \"node-resolver-cdf94\" (UID: \"90df09a3-8706-4883-b711-be63e8e9f191\") " pod="openshift-dns/node-resolver-cdf94"
Oct 30 18:35:59.390572 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.389998    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92f38b64-26fb-42d6-826a-94cd37e8b668-original-pull-secret\") pod \"global-pull-secret-syncer-vs8lx\" (UID: \"92f38b64-26fb-42d6-826a-94cd37e8b668\") " pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:35:59.390572 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390027    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-multus-conf-dir\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.390572 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390101    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-multus-conf-dir\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.390572 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390103    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b9849d01-613f-439d-86de-b517ee4a3341-serviceca\") pod \"node-ca-vtp2j\" (UID: \"b9849d01-613f-439d-86de-b517ee4a3341\") " pod="openshift-image-registry/node-ca-vtp2j"
Oct 30 18:35:59.390572 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390164    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92f38b64-26fb-42d6-826a-94cd37e8b668-dbus\") pod \"global-pull-secret-syncer-vs8lx\" (UID: \"92f38b64-26fb-42d6-826a-94cd37e8b668\") " pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:35:59.390572 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390211    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e0da5dc-3a50-414f-99aa-0a8bbe303a31-agent-certs\") pod \"konnectivity-agent-5v2r9\" (UID: \"1e0da5dc-3a50-414f-99aa-0a8bbe303a31\") " pod="kube-system/konnectivity-agent-5v2r9"
Oct 30 18:35:59.390572 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390252    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-multus-cni-dir\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.390572 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390272    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-modprobe-d\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.390572 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390298    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-socket-dir\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:35:59.390572 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390316    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65beb5a5-1df3-42c8-9e14-769b4ad47255-cni-binary-copy\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:35:59.390572 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390349    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1c10266-f60d-4b43-b2ee-cef69669bc7b-ovnkube-script-lib\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.390572 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390355    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e0da5dc-3a50-414f-99aa-0a8bbe303a31-konnectivity-ca\") pod \"konnectivity-agent-5v2r9\" (UID: \"1e0da5dc-3a50-414f-99aa-0a8bbe303a31\") " pod="kube-system/konnectivity-agent-5v2r9"
Oct 30 18:35:59.390572 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390376    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-var-lib-kubelet\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.390572 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390397    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-run-multus-certs\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.390572 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390416    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-var-lib-kubelet\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.390572 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390424    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-sysconfig\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.390572 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390461    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs\") pod \"network-metrics-daemon-sqkfq\" (UID: \"6df0f1e7-07b7-4231-bc70-ba81aef29856\") " pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:35:59.391367 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390477    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-multus-cni-dir\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.391367 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390483    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-run-systemd\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.391367 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390492    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-run-multus-certs\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.391367 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390524    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1c10266-f60d-4b43-b2ee-cef69669bc7b-ovnkube-config\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.391367 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390540    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1c10266-f60d-4b43-b2ee-cef69669bc7b-ovn-node-metrics-cert\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.391367 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390601    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65beb5a5-1df3-42c8-9e14-769b4ad47255-system-cni-dir\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:35:59.391367 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390607    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-run-systemd\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.391367 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390644    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65beb5a5-1df3-42c8-9e14-769b4ad47255-cnibin\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:35:59.391367 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390693    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-system-cni-dir\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.391367 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390723    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-multus-socket-dir-parent\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.391367 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390747    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-log-socket\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.391367 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390757    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-system-cni-dir\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.391367 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390773    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb856\" (UniqueName: \"kubernetes.io/projected/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278-kube-api-access-vb856\") pod \"network-check-target-xcmps\" (UID: \"0b0aa783-d4ab-490c-a5f7-d7da5ad8d278\") " pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:35:59.391367 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390789    2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 18:35:59.391367 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390800    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-run-k8s-cni-cncf-io\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.391367 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390795    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-multus-socket-dir-parent\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.391367 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390801    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-log-socket\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.391367 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390829    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5c5k\" (UniqueName: \"kubernetes.io/projected/f416f20b-a466-4f3d-bd2e-bb825b189067-kube-api-access-m5c5k\") pod \"iptables-alerter-44cfg\" (UID: \"f416f20b-a466-4f3d-bd2e-bb825b189067\") " pod="openshift-network-operator/iptables-alerter-44cfg"
Oct 30 18:35:59.392789 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390855    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-sys-fs\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:35:59.392789 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390871    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-run-k8s-cni-cncf-io\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.392789 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390881    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90df09a3-8706-4883-b711-be63e8e9f191-hosts-file\") pod \"node-resolver-cdf94\" (UID: \"90df09a3-8706-4883-b711-be63e8e9f191\") " pod="openshift-dns/node-resolver-cdf94"
Oct 30 18:35:59.392789 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390925    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvl8c\" (UniqueName: \"kubernetes.io/projected/b9849d01-613f-439d-86de-b517ee4a3341-kube-api-access-jvl8c\") pod \"node-ca-vtp2j\" (UID: \"b9849d01-613f-439d-86de-b517ee4a3341\") " pod="openshift-image-registry/node-ca-vtp2j"
Oct 30 18:35:59.392789 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.390966    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1c10266-f60d-4b43-b2ee-cef69669bc7b-env-overrides\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.392789 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391015    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/74a85732-6555-48bf-9ece-4e863cd1fb93-multus-daemon-config\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.392789 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391019    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1c10266-f60d-4b43-b2ee-cef69669bc7b-ovnkube-config\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.392789 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391105    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-etc-kubernetes\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.392789 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391131    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkl9j\" (UniqueName: \"kubernetes.io/projected/74a85732-6555-48bf-9ece-4e863cd1fb93-kube-api-access-qkl9j\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.392789 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391148    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-sysctl-conf\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.392789 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391163    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-etc-kubernetes\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.392789 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391164    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f416f20b-a466-4f3d-bd2e-bb825b189067-iptables-alerter-script\") pod \"iptables-alerter-44cfg\" (UID: \"f416f20b-a466-4f3d-bd2e-bb825b189067\") " pod="openshift-network-operator/iptables-alerter-44cfg"
Oct 30 18:35:59.392789 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391220    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpqmj\" (UniqueName: \"kubernetes.io/projected/65beb5a5-1df3-42c8-9e14-769b4ad47255-kube-api-access-jpqmj\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:35:59.392789 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391265    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-registration-dir\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:35:59.392789 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391290    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-run-ovn\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.392789 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391318    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-run-ovn\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.392789 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391340    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-cni-netd\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.393298 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391356    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-sys\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.393298 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391373    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-var-lib-kubelet\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.393298 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391375    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1c10266-f60d-4b43-b2ee-cef69669bc7b-env-overrides\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.393298 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391393    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3935b0fd-69be-4272-adc5-ecfbc7af04ae-tmp\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.393298 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391416    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-cni-netd\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.393298 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391436    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tsxg\" (UniqueName: \"kubernetes.io/projected/e1c10266-f60d-4b43-b2ee-cef69669bc7b-kube-api-access-4tsxg\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.393298 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391493    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74a85732-6555-48bf-9ece-4e863cd1fb93-cni-binary-copy\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.393298 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391512    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-run-netns\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.393298 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391529    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-kubernetes\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.393298 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391545    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-sysctl-d\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.393298 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391561    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f416f20b-a466-4f3d-bd2e-bb825b189067-host-slash\") pod \"iptables-alerter-44cfg\" (UID: \"f416f20b-a466-4f3d-bd2e-bb825b189067\") " pod="openshift-network-operator/iptables-alerter-44cfg"
Oct 30 18:35:59.393298 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391584    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-lib-modules\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.393298 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391600    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-etc-selinux\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:35:59.393298 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391646    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-run-netns\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.393298 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391645    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65beb5a5-1df3-42c8-9e14-769b4ad47255-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:35:59.393298 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391684    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65beb5a5-1df3-42c8-9e14-769b4ad47255-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:35:59.393298 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391709    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-run-ovn-kubernetes\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.393784 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391726    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-run-ovn-kubernetes\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.393784 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391748    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.393784 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391769    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-var-lib-cni-bin\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.393784 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391786    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-kubelet-dir\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:35:59.393784 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391802    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-systemd-units\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.393784 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391803    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-var-lib-cni-bin\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.393784 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391821    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-systemd-units\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.393784 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391824    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.393784 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391846    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-cni-bin\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.393784 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391904    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-tuned\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.393784 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391929    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-cni-bin\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.393784 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391961    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fdth\" (UniqueName: \"kubernetes.io/projected/6df0f1e7-07b7-4231-bc70-ba81aef29856-kube-api-access-4fdth\") pod \"network-metrics-daemon-sqkfq\" (UID: \"6df0f1e7-07b7-4231-bc70-ba81aef29856\") " pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:35:59.393784 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.391987    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-device-dir\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:35:59.393784 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.392102    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/74a85732-6555-48bf-9ece-4e863cd1fb93-multus-daemon-config\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.393784 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.392292    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74a85732-6555-48bf-9ece-4e863cd1fb93-cni-binary-copy\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.394390 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.394372    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1c10266-f60d-4b43-b2ee-cef69669bc7b-ovn-node-metrics-cert\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.435464 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.435444    2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 18:35:59.435548 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.435501    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-44bqt"
Oct 30 18:35:59.439294 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.439271    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 18:35:59.439388 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.439329    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:35:59.439388 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.439382    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 18:35:59.439473 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.439434    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 18:35:59.439473 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.439460    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 18:35:59.439843 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.439811    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 18:35:59.439843 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.439824    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 18:35:59.439948 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.439862    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 18:35:59.439948 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.439872    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 18:35:59.439948 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.439873    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 18:35:59.439948 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.439910    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 18:35:59.439948 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.439920    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 18:35:59.440242 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.439961    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 18:35:59.440242 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.439991    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vd777\""
Oct 30 18:35:59.440242 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.440014    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:35:59.440242 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.440018    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 18:35:59.440242 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.440017    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 18:35:59.440242 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.440033    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m6qzp\""
Oct 30 18:35:59.440242 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.440117    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7kzh2\""
Oct 30 18:35:59.440242 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.440116    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 18:35:59.440242 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.440138    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vpn2j\""
Oct 30 18:35:59.440242 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.440228    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h4btl\""
Oct 30 18:35:59.440534 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.440265    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j6hdp\""
Oct 30 18:35:59.452429 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:59.452408    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:35:59.452477 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:59.452432    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:35:59.452477 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:59.452445    2572 projected.go:196] Error preparing data for projected volume kube-api-access-vb856 for pod openshift-network-diagnostics/network-check-target-xcmps: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:35:59.452555 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:59.452515    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278-kube-api-access-vb856 podName:0b0aa783-d4ab-490c-a5f7-d7da5ad8d278 nodeName:}" failed. No retries permitted until 2026-10-30 18:35:59.95249522 +0000 UTC m=+3.132517343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vb856" (UniqueName: "kubernetes.io/projected/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278-kube-api-access-vb856") pod "network-check-target-xcmps" (UID: "0b0aa783-d4ab-490c-a5f7-d7da5ad8d278") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:35:59.454553 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.454532    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkl9j\" (UniqueName: \"kubernetes.io/projected/74a85732-6555-48bf-9ece-4e863cd1fb93-kube-api-access-qkl9j\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.454630 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.454560    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tsxg\" (UniqueName: \"kubernetes.io/projected/e1c10266-f60d-4b43-b2ee-cef69669bc7b-kube-api-access-4tsxg\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.492469 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.492447    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-sys\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.492566 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.492470    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-var-lib-kubelet\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.492566 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.492486    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3935b0fd-69be-4272-adc5-ecfbc7af04ae-tmp\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.492566 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.492501    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-kubernetes\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.492566 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.492515    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-sysctl-d\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.492566 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.492552    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-kubernetes\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.492752 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.492568    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-var-lib-kubelet\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.492752 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.492578    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-sys\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.492752 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.492625    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f416f20b-a466-4f3d-bd2e-bb825b189067-host-slash\") pod \"iptables-alerter-44cfg\" (UID: \"f416f20b-a466-4f3d-bd2e-bb825b189067\") " pod="openshift-network-operator/iptables-alerter-44cfg"
Oct 30 18:35:59.492752 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.492643    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-sysctl-d\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.492752 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.492651    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-lib-modules\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.492752 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.492663    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f416f20b-a466-4f3d-bd2e-bb825b189067-host-slash\") pod \"iptables-alerter-44cfg\" (UID: \"f416f20b-a466-4f3d-bd2e-bb825b189067\") " pod="openshift-network-operator/iptables-alerter-44cfg"
Oct 30 18:35:59.492752 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.492688    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-etc-selinux\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:35:59.492752 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.492705    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65beb5a5-1df3-42c8-9e14-769b4ad47255-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:35:59.492752 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.492728    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65beb5a5-1df3-42c8-9e14-769b4ad47255-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:35:59.493246 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.492755    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-kubelet-dir\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:35:59.493246 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.492784    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-tuned\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.493246 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.492788    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-lib-modules\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.493246 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.492825    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fdth\" (UniqueName: \"kubernetes.io/projected/6df0f1e7-07b7-4231-bc70-ba81aef29856-kube-api-access-4fdth\") pod \"network-metrics-daemon-sqkfq\" (UID: \"6df0f1e7-07b7-4231-bc70-ba81aef29856\") " pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:35:59.493246 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.492828    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-etc-selinux\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:35:59.493246 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.492851    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-device-dir\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:35:59.493246 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.492880    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-device-dir\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:35:59.493246 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.492883    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65beb5a5-1df3-42c8-9e14-769b4ad47255-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:35:59.493246 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.492901    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-kubelet-dir\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:35:59.493246 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.492971    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9849d01-613f-439d-86de-b517ee4a3341-host\") pod \"node-ca-vtp2j\" (UID: \"b9849d01-613f-439d-86de-b517ee4a3341\") " pod="openshift-image-registry/node-ca-vtp2j"
Oct 30 18:35:59.493246 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.492991    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-systemd\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.493246 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493005    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-run\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.493246 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493023    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9849d01-613f-439d-86de-b517ee4a3341-host\") pod \"node-ca-vtp2j\" (UID: \"b9849d01-613f-439d-86de-b517ee4a3341\") " pod="openshift-image-registry/node-ca-vtp2j"
Oct 30 18:35:59.493246 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493042    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-run\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.493246 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493048    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-systemd\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.493246 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493090    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhnjp\" (UniqueName: \"kubernetes.io/projected/90df09a3-8706-4883-b711-be63e8e9f191-kube-api-access-hhnjp\") pod \"node-resolver-cdf94\" (UID: \"90df09a3-8706-4883-b711-be63e8e9f191\") " pod="openshift-dns/node-resolver-cdf94"
Oct 30 18:35:59.493246 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493120    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92f38b64-26fb-42d6-826a-94cd37e8b668-kubelet-config\") pod \"global-pull-secret-syncer-vs8lx\" (UID: \"92f38b64-26fb-42d6-826a-94cd37e8b668\") " pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:35:59.494170 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493152    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-host\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.494170 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493160    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92f38b64-26fb-42d6-826a-94cd37e8b668-kubelet-config\") pod \"global-pull-secret-syncer-vs8lx\" (UID: \"92f38b64-26fb-42d6-826a-94cd37e8b668\") " pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:35:59.494170 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493191    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6bwn\" (UniqueName: \"kubernetes.io/projected/3935b0fd-69be-4272-adc5-ecfbc7af04ae-kube-api-access-g6bwn\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.494170 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493217    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p6bb\" (UniqueName: \"kubernetes.io/projected/9092fa02-cc22-41a2-b4ee-070298ecab99-kube-api-access-4p6bb\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:35:59.494170 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493242    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-host\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.494170 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493243    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65beb5a5-1df3-42c8-9e14-769b4ad47255-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:35:59.494170 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493283    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90df09a3-8706-4883-b711-be63e8e9f191-tmp-dir\") pod \"node-resolver-cdf94\" (UID: \"90df09a3-8706-4883-b711-be63e8e9f191\") " pod="openshift-dns/node-resolver-cdf94"
Oct 30 18:35:59.494170 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493310    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92f38b64-26fb-42d6-826a-94cd37e8b668-original-pull-secret\") pod \"global-pull-secret-syncer-vs8lx\" (UID: \"92f38b64-26fb-42d6-826a-94cd37e8b668\") " pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:35:59.494170 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493341    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b9849d01-613f-439d-86de-b517ee4a3341-serviceca\") pod \"node-ca-vtp2j\" (UID: \"b9849d01-613f-439d-86de-b517ee4a3341\") " pod="openshift-image-registry/node-ca-vtp2j"
Oct 30 18:35:59.494170 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493365    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92f38b64-26fb-42d6-826a-94cd37e8b668-dbus\") pod \"global-pull-secret-syncer-vs8lx\" (UID: \"92f38b64-26fb-42d6-826a-94cd37e8b668\") " pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:35:59.494170 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493391    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e0da5dc-3a50-414f-99aa-0a8bbe303a31-agent-certs\") pod \"konnectivity-agent-5v2r9\" (UID: \"1e0da5dc-3a50-414f-99aa-0a8bbe303a31\") " pod="kube-system/konnectivity-agent-5v2r9"
Oct 30 18:35:59.494170 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493394    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65beb5a5-1df3-42c8-9e14-769b4ad47255-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:35:59.494170 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493417    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-modprobe-d\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.494170 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493444    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-socket-dir\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:35:59.494170 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493471    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65beb5a5-1df3-42c8-9e14-769b4ad47255-cni-binary-copy\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:35:59.494170 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493498    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e0da5dc-3a50-414f-99aa-0a8bbe303a31-konnectivity-ca\") pod \"konnectivity-agent-5v2r9\" (UID: \"1e0da5dc-3a50-414f-99aa-0a8bbe303a31\") " pod="kube-system/konnectivity-agent-5v2r9"
Oct 30 18:35:59.494170 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493530    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-sysconfig\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.494922 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493558    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs\") pod \"network-metrics-daemon-sqkfq\" (UID: \"6df0f1e7-07b7-4231-bc70-ba81aef29856\") " pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:35:59.494922 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493588    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65beb5a5-1df3-42c8-9e14-769b4ad47255-system-cni-dir\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:35:59.494922 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493617    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65beb5a5-1df3-42c8-9e14-769b4ad47255-cnibin\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:35:59.494922 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493671    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5c5k\" (UniqueName: \"kubernetes.io/projected/f416f20b-a466-4f3d-bd2e-bb825b189067-kube-api-access-m5c5k\") pod \"iptables-alerter-44cfg\" (UID: \"f416f20b-a466-4f3d-bd2e-bb825b189067\") " pod="openshift-network-operator/iptables-alerter-44cfg"
Oct 30 18:35:59.494922 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493699    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-sys-fs\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:35:59.494922 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:59.493730    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:35:59.494922 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493748    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90df09a3-8706-4883-b711-be63e8e9f191-hosts-file\") pod \"node-resolver-cdf94\" (UID: \"90df09a3-8706-4883-b711-be63e8e9f191\") " pod="openshift-dns/node-resolver-cdf94"
Oct 30 18:35:59.494922 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493747    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65beb5a5-1df3-42c8-9e14-769b4ad47255-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:35:59.494922 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493784    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90df09a3-8706-4883-b711-be63e8e9f191-tmp-dir\") pod \"node-resolver-cdf94\" (UID: \"90df09a3-8706-4883-b711-be63e8e9f191\") " pod="openshift-dns/node-resolver-cdf94"
Oct 30 18:35:59.494922 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:59.493805    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92f38b64-26fb-42d6-826a-94cd37e8b668-original-pull-secret podName:92f38b64-26fb-42d6-826a-94cd37e8b668 nodeName:}" failed. No retries permitted until 2026-10-30 18:35:59.993771569 +0000 UTC m=+3.173793700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92f38b64-26fb-42d6-826a-94cd37e8b668-original-pull-secret") pod "global-pull-secret-syncer-vs8lx" (UID: "92f38b64-26fb-42d6-826a-94cd37e8b668") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:35:59.494922 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493836    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-sysconfig\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.494922 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493843    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b9849d01-613f-439d-86de-b517ee4a3341-serviceca\") pod \"node-ca-vtp2j\" (UID: \"b9849d01-613f-439d-86de-b517ee4a3341\") " pod="openshift-image-registry/node-ca-vtp2j"
Oct 30 18:35:59.494922 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493886    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65beb5a5-1df3-42c8-9e14-769b4ad47255-cnibin\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:35:59.494922 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493913    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-modprobe-d\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.494922 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:59.493938    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:35:59.494922 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493942    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65beb5a5-1df3-42c8-9e14-769b4ad47255-system-cni-dir\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:35:59.494922 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.493978    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92f38b64-26fb-42d6-826a-94cd37e8b668-dbus\") pod \"global-pull-secret-syncer-vs8lx\" (UID: \"92f38b64-26fb-42d6-826a-94cd37e8b668\") " pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:35:59.495551 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:59.493987    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs podName:6df0f1e7-07b7-4231-bc70-ba81aef29856 nodeName:}" failed. No retries permitted until 2026-10-30 18:35:59.993970971 +0000 UTC m=+3.173993083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs") pod "network-metrics-daemon-sqkfq" (UID: "6df0f1e7-07b7-4231-bc70-ba81aef29856") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:35:59.495551 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.494004    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-sys-fs\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:35:59.495551 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.494086    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90df09a3-8706-4883-b711-be63e8e9f191-hosts-file\") pod \"node-resolver-cdf94\" (UID: \"90df09a3-8706-4883-b711-be63e8e9f191\") " pod="openshift-dns/node-resolver-cdf94"
Oct 30 18:35:59.495551 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.494096    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-socket-dir\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:35:59.495551 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.494124    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvl8c\" (UniqueName: \"kubernetes.io/projected/b9849d01-613f-439d-86de-b517ee4a3341-kube-api-access-jvl8c\") pod \"node-ca-vtp2j\" (UID: \"b9849d01-613f-439d-86de-b517ee4a3341\") " pod="openshift-image-registry/node-ca-vtp2j"
Oct 30 18:35:59.495551 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.494183    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-sysctl-conf\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.495551 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.494213    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f416f20b-a466-4f3d-bd2e-bb825b189067-iptables-alerter-script\") pod \"iptables-alerter-44cfg\" (UID: \"f416f20b-a466-4f3d-bd2e-bb825b189067\") " pod="openshift-network-operator/iptables-alerter-44cfg"
Oct 30 18:35:59.495551 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.494241    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpqmj\" (UniqueName: \"kubernetes.io/projected/65beb5a5-1df3-42c8-9e14-769b4ad47255-kube-api-access-jpqmj\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:35:59.495551 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.494266    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-registration-dir\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:35:59.495551 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.494265    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e0da5dc-3a50-414f-99aa-0a8bbe303a31-konnectivity-ca\") pod \"konnectivity-agent-5v2r9\" (UID: \"1e0da5dc-3a50-414f-99aa-0a8bbe303a31\") " pod="kube-system/konnectivity-agent-5v2r9"
Oct 30 18:35:59.495551 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.494450    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-sysctl-conf\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.495551 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.494516    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-registration-dir\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:35:59.495551 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.494527    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65beb5a5-1df3-42c8-9e14-769b4ad47255-cni-binary-copy\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:35:59.495551 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.494746    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f416f20b-a466-4f3d-bd2e-bb825b189067-iptables-alerter-script\") pod \"iptables-alerter-44cfg\" (UID: \"f416f20b-a466-4f3d-bd2e-bb825b189067\") " pod="openshift-network-operator/iptables-alerter-44cfg"
Oct 30 18:35:59.495551 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.495139    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3935b0fd-69be-4272-adc5-ecfbc7af04ae-tmp\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.495551 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.495207    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-tuned\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.496228 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.496212    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e0da5dc-3a50-414f-99aa-0a8bbe303a31-agent-certs\") pod \"konnectivity-agent-5v2r9\" (UID: \"1e0da5dc-3a50-414f-99aa-0a8bbe303a31\") " pod="kube-system/konnectivity-agent-5v2r9"
Oct 30 18:35:59.557169 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.557108    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p6bb\" (UniqueName: \"kubernetes.io/projected/9092fa02-cc22-41a2-b4ee-070298ecab99-kube-api-access-4p6bb\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:35:59.557169 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.557121    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6bwn\" (UniqueName: \"kubernetes.io/projected/3935b0fd-69be-4272-adc5-ecfbc7af04ae-kube-api-access-g6bwn\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.557840 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.557814    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvl8c\" (UniqueName: \"kubernetes.io/projected/b9849d01-613f-439d-86de-b517ee4a3341-kube-api-access-jvl8c\") pod \"node-ca-vtp2j\" (UID: \"b9849d01-613f-439d-86de-b517ee4a3341\") " pod="openshift-image-registry/node-ca-vtp2j"
Oct 30 18:35:59.558069 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.558044    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5c5k\" (UniqueName: \"kubernetes.io/projected/f416f20b-a466-4f3d-bd2e-bb825b189067-kube-api-access-m5c5k\") pod \"iptables-alerter-44cfg\" (UID: \"f416f20b-a466-4f3d-bd2e-bb825b189067\") " pod="openshift-network-operator/iptables-alerter-44cfg"
Oct 30 18:35:59.558981 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.558961    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpqmj\" (UniqueName: \"kubernetes.io/projected/65beb5a5-1df3-42c8-9e14-769b4ad47255-kube-api-access-jpqmj\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:35:59.559116 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.559097    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhnjp\" (UniqueName: \"kubernetes.io/projected/90df09a3-8706-4883-b711-be63e8e9f191-kube-api-access-hhnjp\") pod \"node-resolver-cdf94\" (UID: \"90df09a3-8706-4883-b711-be63e8e9f191\") " pod="openshift-dns/node-resolver-cdf94"
Oct 30 18:35:59.559168 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.559133    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fdth\" (UniqueName: \"kubernetes.io/projected/6df0f1e7-07b7-4231-bc70-ba81aef29856-kube-api-access-4fdth\") pod \"network-metrics-daemon-sqkfq\" (UID: \"6df0f1e7-07b7-4231-bc70-ba81aef29856\") " pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:35:59.584238 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.584222    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wd7km"
Oct 30 18:35:59.594328 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:59.594300    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a85732_6555_48bf_9ece_4e863cd1fb93.slice/crio-90724e59845a496535c3fa041fad3cb95848b518bd1225c142dd012354e9e99e WatchSource:0}: Error finding container 90724e59845a496535c3fa041fad3cb95848b518bd1225c142dd012354e9e99e: Status 404 returned error can't find the container with id 90724e59845a496535c3fa041fad3cb95848b518bd1225c142dd012354e9e99e
Oct 30 18:35:59.602243 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.602204    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-44cfg"
Oct 30 18:35:59.610465 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:59.610440    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf416f20b_a466_4f3d_bd2e_bb825b189067.slice/crio-80977e6181a3c742059b0adae7b88dbd4190d264ac3da7c81df5c9a38370a97d WatchSource:0}: Error finding container 80977e6181a3c742059b0adae7b88dbd4190d264ac3da7c81df5c9a38370a97d: Status 404 returned error can't find the container with id 80977e6181a3c742059b0adae7b88dbd4190d264ac3da7c81df5c9a38370a97d
Oct 30 18:35:59.615659 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.615641    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5v2r9"
Oct 30 18:35:59.616816 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.616799    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:35:59.619536 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.619516    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:35:59.621785 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.621700    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:35:59.624555 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:59.624537    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e0da5dc_3a50_414f_99aa_0a8bbe303a31.slice/crio-d42e55117f9bd1279beabb43c4a75ceb656285a170dffb6bdc9e7531b4666440 WatchSource:0}: Error finding container d42e55117f9bd1279beabb43c4a75ceb656285a170dffb6bdc9e7531b4666440: Status 404 returned error can't find the container with id d42e55117f9bd1279beabb43c4a75ceb656285a170dffb6bdc9e7531b4666440
Oct 30 18:35:59.627907 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:59.627861    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1c10266_f60d_4b43_b2ee_cef69669bc7b.slice/crio-f863f9d519de3c1bdafab1e8717dd8731b7fafdd7c9c7b23a372d5d64b18c521 WatchSource:0}: Error finding container f863f9d519de3c1bdafab1e8717dd8731b7fafdd7c9c7b23a372d5d64b18c521: Status 404 returned error can't find the container with id f863f9d519de3c1bdafab1e8717dd8731b7fafdd7c9c7b23a372d5d64b18c521
Oct 30 18:35:59.631635 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:59.631567    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9092fa02_cc22_41a2_b4ee_070298ecab99.slice/crio-47de5c3c76d6b584d3f74b82db93dff272d550428be9da41ddfe3619fa3ec2d5 WatchSource:0}: Error finding container 47de5c3c76d6b584d3f74b82db93dff272d550428be9da41ddfe3619fa3ec2d5: Status 404 returned error can't find the container with id 47de5c3c76d6b584d3f74b82db93dff272d550428be9da41ddfe3619fa3ec2d5
Oct 30 18:35:59.633481 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:59.633462    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3935b0fd_69be_4272_adc5_ecfbc7af04ae.slice/crio-28a085f4f5b01a4d8d660f75fbea980faeb2e4a471c11b983d659513fe241b1d WatchSource:0}: Error finding container 28a085f4f5b01a4d8d660f75fbea980faeb2e4a471c11b983d659513fe241b1d: Status 404 returned error can't find the container with id 28a085f4f5b01a4d8d660f75fbea980faeb2e4a471c11b983d659513fe241b1d
Oct 30 18:35:59.634504 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.634474    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cdf94"
Oct 30 18:35:59.637985 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.637966    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vtp2j"
Oct 30 18:35:59.641839 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.641819    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:35:59.644345 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:59.644322    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90df09a3_8706_4883_b711_be63e8e9f191.slice/crio-5a5a5bcab19b8f9cd4474e125f5e0e25e6671882e7c8e722fbeb629c8f8f3821 WatchSource:0}: Error finding container 5a5a5bcab19b8f9cd4474e125f5e0e25e6671882e7c8e722fbeb629c8f8f3821: Status 404 returned error can't find the container with id 5a5a5bcab19b8f9cd4474e125f5e0e25e6671882e7c8e722fbeb629c8f8f3821
Oct 30 18:35:59.648129 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:59.648097    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9849d01_613f_439d_86de_b517ee4a3341.slice/crio-f9746669de1d07f5c2c427a3f47b253acf5cab3b53fd2d2afe6019626e24c8ba WatchSource:0}: Error finding container f9746669de1d07f5c2c427a3f47b253acf5cab3b53fd2d2afe6019626e24c8ba: Status 404 returned error can't find the container with id f9746669de1d07f5c2c427a3f47b253acf5cab3b53fd2d2afe6019626e24c8ba
Oct 30 18:35:59.650989 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:35:59.650971    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65beb5a5_1df3_42c8_9e14_769b4ad47255.slice/crio-b5e736af7cb22e7e295d2c82ed4c7548f4498a44bbd419920772c5a90e05ce4e WatchSource:0}: Error finding container b5e736af7cb22e7e295d2c82ed4c7548f4498a44bbd419920772c5a90e05ce4e: Status 404 returned error can't find the container with id b5e736af7cb22e7e295d2c82ed4c7548f4498a44bbd419920772c5a90e05ce4e
Oct 30 18:35:59.997362 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.997279    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92f38b64-26fb-42d6-826a-94cd37e8b668-original-pull-secret\") pod \"global-pull-secret-syncer-vs8lx\" (UID: \"92f38b64-26fb-42d6-826a-94cd37e8b668\") " pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:35:59.997362 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.997334    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs\") pod \"network-metrics-daemon-sqkfq\" (UID: \"6df0f1e7-07b7-4231-bc70-ba81aef29856\") " pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:35:59.997591 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:35:59.997367    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb856\" (UniqueName: \"kubernetes.io/projected/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278-kube-api-access-vb856\") pod \"network-check-target-xcmps\" (UID: \"0b0aa783-d4ab-490c-a5f7-d7da5ad8d278\") " pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:35:59.997591 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:59.997439    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:35:59.997591 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:59.997453    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:35:59.997591 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:59.997468    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:35:59.997591 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:59.997484    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:35:59.997591 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:59.997496    2572 projected.go:196] Error preparing data for projected volume kube-api-access-vb856 for pod openshift-network-diagnostics/network-check-target-xcmps: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:35:59.997591 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:59.997507    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92f38b64-26fb-42d6-826a-94cd37e8b668-original-pull-secret podName:92f38b64-26fb-42d6-826a-94cd37e8b668 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:00.997487549 +0000 UTC m=+4.177509663 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92f38b64-26fb-42d6-826a-94cd37e8b668-original-pull-secret") pod "global-pull-secret-syncer-vs8lx" (UID: "92f38b64-26fb-42d6-826a-94cd37e8b668") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:35:59.997591 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:59.997525    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs podName:6df0f1e7-07b7-4231-bc70-ba81aef29856 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:00.997515548 +0000 UTC m=+4.177537655 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs") pod "network-metrics-daemon-sqkfq" (UID: "6df0f1e7-07b7-4231-bc70-ba81aef29856") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:35:59.997591 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:35:59.997541    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278-kube-api-access-vb856 podName:0b0aa783-d4ab-490c-a5f7-d7da5ad8d278 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:00.997532166 +0000 UTC m=+4.177554281 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vb856" (UniqueName: "kubernetes.io/projected/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278-kube-api-access-vb856") pod "network-check-target-xcmps" (UID: "0b0aa783-d4ab-490c-a5f7-d7da5ad8d278") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:00.317213 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:00.317125    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vtp2j" event={"ID":"b9849d01-613f-439d-86de-b517ee4a3341","Type":"ContainerStarted","Data":"f9746669de1d07f5c2c427a3f47b253acf5cab3b53fd2d2afe6019626e24c8ba"}
Oct 30 18:36:00.318597 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:00.318569    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bhxxm" event={"ID":"3935b0fd-69be-4272-adc5-ecfbc7af04ae","Type":"ContainerStarted","Data":"28a085f4f5b01a4d8d660f75fbea980faeb2e4a471c11b983d659513fe241b1d"}
Oct 30 18:36:00.320464 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:00.320396    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz" event={"ID":"9092fa02-cc22-41a2-b4ee-070298ecab99","Type":"ContainerStarted","Data":"47de5c3c76d6b584d3f74b82db93dff272d550428be9da41ddfe3619fa3ec2d5"}
Oct 30 18:36:00.321961 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:00.321932    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r" event={"ID":"e1c10266-f60d-4b43-b2ee-cef69669bc7b","Type":"ContainerStarted","Data":"f863f9d519de3c1bdafab1e8717dd8731b7fafdd7c9c7b23a372d5d64b18c521"}
Oct 30 18:36:00.323369 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:00.323338    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5v2r9" event={"ID":"1e0da5dc-3a50-414f-99aa-0a8bbe303a31","Type":"ContainerStarted","Data":"d42e55117f9bd1279beabb43c4a75ceb656285a170dffb6bdc9e7531b4666440"}
Oct 30 18:36:00.324742 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:00.324713    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wd7km" event={"ID":"74a85732-6555-48bf-9ece-4e863cd1fb93","Type":"ContainerStarted","Data":"90724e59845a496535c3fa041fad3cb95848b518bd1225c142dd012354e9e99e"}
Oct 30 18:36:00.326026 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:00.325975    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h44bp" event={"ID":"65beb5a5-1df3-42c8-9e14-769b4ad47255","Type":"ContainerStarted","Data":"b5e736af7cb22e7e295d2c82ed4c7548f4498a44bbd419920772c5a90e05ce4e"}
Oct 30 18:36:00.327269 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:00.327246    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cdf94" event={"ID":"90df09a3-8706-4883-b711-be63e8e9f191","Type":"ContainerStarted","Data":"5a5a5bcab19b8f9cd4474e125f5e0e25e6671882e7c8e722fbeb629c8f8f3821"}
Oct 30 18:36:00.328568 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:00.328543    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-44cfg" event={"ID":"f416f20b-a466-4f3d-bd2e-bb825b189067","Type":"ContainerStarted","Data":"80977e6181a3c742059b0adae7b88dbd4190d264ac3da7c81df5c9a38370a97d"}
Oct 30 18:36:00.440463 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:00.440359    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:30:59 +0000 UTC" deadline="2028-05-28 10:05:10.79813447 +0000 UTC"
Oct 30 18:36:00.440463 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:00.440413    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13815h29m10.357726501s"
Oct 30 18:36:01.007659 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:01.007611    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs\") pod \"network-metrics-daemon-sqkfq\" (UID: \"6df0f1e7-07b7-4231-bc70-ba81aef29856\") " pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:01.007855 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:01.007685    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb856\" (UniqueName: \"kubernetes.io/projected/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278-kube-api-access-vb856\") pod \"network-check-target-xcmps\" (UID: \"0b0aa783-d4ab-490c-a5f7-d7da5ad8d278\") " pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:36:01.007855 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:01.007847    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:36:01.007984 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:01.007868    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:36:01.007984 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:01.007883    2572 projected.go:196] Error preparing data for projected volume kube-api-access-vb856 for pod openshift-network-diagnostics/network-check-target-xcmps: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:01.007984 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:01.007950    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278-kube-api-access-vb856 podName:0b0aa783-d4ab-490c-a5f7-d7da5ad8d278 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:03.007930528 +0000 UTC m=+6.187952658 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vb856" (UniqueName: "kubernetes.io/projected/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278-kube-api-access-vb856") pod "network-check-target-xcmps" (UID: "0b0aa783-d4ab-490c-a5f7-d7da5ad8d278") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:01.008168 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:01.008037    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:01.008168 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:01.008077    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs podName:6df0f1e7-07b7-4231-bc70-ba81aef29856 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:03.008064619 +0000 UTC m=+6.188086740 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs") pod "network-metrics-daemon-sqkfq" (UID: "6df0f1e7-07b7-4231-bc70-ba81aef29856") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:01.008281 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:01.008181    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92f38b64-26fb-42d6-826a-94cd37e8b668-original-pull-secret\") pod \"global-pull-secret-syncer-vs8lx\" (UID: \"92f38b64-26fb-42d6-826a-94cd37e8b668\") " pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:01.008281 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:01.008271    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:01.008381 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:01.008314    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92f38b64-26fb-42d6-826a-94cd37e8b668-original-pull-secret podName:92f38b64-26fb-42d6-826a-94cd37e8b668 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:03.008301528 +0000 UTC m=+6.188323636 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92f38b64-26fb-42d6-826a-94cd37e8b668-original-pull-secret") pod "global-pull-secret-syncer-vs8lx" (UID: "92f38b64-26fb-42d6-826a-94cd37e8b668") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:01.311273 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:01.311197    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:01.311434 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:01.311353    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqkfq" podUID="6df0f1e7-07b7-4231-bc70-ba81aef29856"
Oct 30 18:36:01.311502 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:01.311453    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:36:01.311560 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:01.311534    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xcmps" podUID="0b0aa783-d4ab-490c-a5f7-d7da5ad8d278"
Oct 30 18:36:01.311613 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:01.311601    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:01.311693 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:01.311673    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8lx" podUID="92f38b64-26fb-42d6-826a-94cd37e8b668"
Oct 30 18:36:01.442243 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:01.442157    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:30:59 +0000 UTC" deadline="2028-04-14 23:42:05.485600649 +0000 UTC"
Oct 30 18:36:01.442243 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:01.442241    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12773h6m4.043363172s"
Oct 30 18:36:02.355313 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:02.355037    2572 generic.go:356] "Generic (PLEG): container finished" podID="0bdd5feb35ab75fe20b4acd57c05f061" containerID="9f10cc8efd5fe47ce08be827a22667753c7145bd11db93cfc05a9814b0c04f67" exitCode=0
Oct 30 18:36:02.355511 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:02.355339    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal" event={"ID":"0bdd5feb35ab75fe20b4acd57c05f061","Type":"ContainerDied","Data":"9f10cc8efd5fe47ce08be827a22667753c7145bd11db93cfc05a9814b0c04f67"}
Oct 30 18:36:03.027389 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:03.027304    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs\") pod \"network-metrics-daemon-sqkfq\" (UID: \"6df0f1e7-07b7-4231-bc70-ba81aef29856\") " pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:03.027389 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:03.027370    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb856\" (UniqueName: \"kubernetes.io/projected/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278-kube-api-access-vb856\") pod \"network-check-target-xcmps\" (UID: \"0b0aa783-d4ab-490c-a5f7-d7da5ad8d278\") " pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:36:03.028020 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:03.027442    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92f38b64-26fb-42d6-826a-94cd37e8b668-original-pull-secret\") pod \"global-pull-secret-syncer-vs8lx\" (UID: \"92f38b64-26fb-42d6-826a-94cd37e8b668\") " pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:03.028020 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:03.027575    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:03.028020 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:03.027644    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92f38b64-26fb-42d6-826a-94cd37e8b668-original-pull-secret podName:92f38b64-26fb-42d6-826a-94cd37e8b668 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:07.027624654 +0000 UTC m=+10.207646760 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92f38b64-26fb-42d6-826a-94cd37e8b668-original-pull-secret") pod "global-pull-secret-syncer-vs8lx" (UID: "92f38b64-26fb-42d6-826a-94cd37e8b668") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:03.028220 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:03.028063    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:03.028220 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:03.028119    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs podName:6df0f1e7-07b7-4231-bc70-ba81aef29856 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:07.028102045 +0000 UTC m=+10.208124152 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs") pod "network-metrics-daemon-sqkfq" (UID: "6df0f1e7-07b7-4231-bc70-ba81aef29856") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:03.028220 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:03.028215    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:36:03.028406 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:03.028234    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:36:03.028406 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:03.028249    2572 projected.go:196] Error preparing data for projected volume kube-api-access-vb856 for pod openshift-network-diagnostics/network-check-target-xcmps: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:03.028406 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:03.028289    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278-kube-api-access-vb856 podName:0b0aa783-d4ab-490c-a5f7-d7da5ad8d278 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:07.028276723 +0000 UTC m=+10.208298823 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vb856" (UniqueName: "kubernetes.io/projected/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278-kube-api-access-vb856") pod "network-check-target-xcmps" (UID: "0b0aa783-d4ab-490c-a5f7-d7da5ad8d278") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:03.307175 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:03.306484    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:36:03.307175 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:03.306630    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xcmps" podUID="0b0aa783-d4ab-490c-a5f7-d7da5ad8d278"
Oct 30 18:36:03.307175 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:03.306692    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:03.307175 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:03.306764    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8lx" podUID="92f38b64-26fb-42d6-826a-94cd37e8b668"
Oct 30 18:36:03.307175 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:03.306812    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:03.307175 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:03.306895    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqkfq" podUID="6df0f1e7-07b7-4231-bc70-ba81aef29856"
Oct 30 18:36:03.362344 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:03.362309    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal" event={"ID":"2e58124d8b2f46cf7f8af1e433d1d9a6","Type":"ContainerStarted","Data":"b71c61a669e677a3e50226c5b94e93d4c59bb83d095bba29d7e925ab298fe447"}
Oct 30 18:36:03.368720 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:03.368692    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal" event={"ID":"0bdd5feb35ab75fe20b4acd57c05f061","Type":"ContainerStarted","Data":"27f84e9cd8fc24fcfa9e906f78856183d3215cbf13b6b115e616a208c1e7a1c8"}
Oct 30 18:36:03.614921 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:03.614805    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal" podStartSLOduration=4.6147828539999995 podStartE2EDuration="4.614782854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:03.48978615 +0000 UTC m=+6.669808274" watchObservedRunningTime="2026-10-30 18:36:03.614782854 +0000 UTC m=+6.794804982"
Oct 30 18:36:04.374339 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:04.374274    2572 generic.go:356] "Generic (PLEG): container finished" podID="65beb5a5-1df3-42c8-9e14-769b4ad47255" containerID="ef49d038bacffd3b76c7e60e66730b18de394d2a31e9ae50b88f893ea54ed914" exitCode=0
Oct 30 18:36:04.374671 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:04.374570    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h44bp" event={"ID":"65beb5a5-1df3-42c8-9e14-769b4ad47255","Type":"ContainerDied","Data":"ef49d038bacffd3b76c7e60e66730b18de394d2a31e9ae50b88f893ea54ed914"}
Oct 30 18:36:04.617855 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:04.616944    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal" podStartSLOduration=5.6169241549999995 podStartE2EDuration="5.616924155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:03.621464569 +0000 UTC m=+6.801486695" watchObservedRunningTime="2026-10-30 18:36:04.616924155 +0000 UTC m=+7.796946280"
Oct 30 18:36:05.310734 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:05.307189    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:05.310734 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:05.307344    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqkfq" podUID="6df0f1e7-07b7-4231-bc70-ba81aef29856"
Oct 30 18:36:05.310734 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:05.307755    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:36:05.310734 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:05.307845    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xcmps" podUID="0b0aa783-d4ab-490c-a5f7-d7da5ad8d278"
Oct 30 18:36:05.310734 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:05.307904    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:05.310734 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:05.307988    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8lx" podUID="92f38b64-26fb-42d6-826a-94cd37e8b668"
Oct 30 18:36:05.383798 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:05.383101    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5v2r9" event={"ID":"1e0da5dc-3a50-414f-99aa-0a8bbe303a31","Type":"ContainerStarted","Data":"cd4c30a12c29a8203d04269622354b2b41d08a8fe5c8de6841012c6e72956f61"}
Oct 30 18:36:05.972367 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:05.972339    2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5v2r9"
Oct 30 18:36:05.972877 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:05.972859    2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5v2r9"
Oct 30 18:36:06.088372 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:06.088274    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5v2r9" podStartSLOduration=3.406801867 podStartE2EDuration="8.088254524s" totalImagesPullingTime="4.681452657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:05.510478664 +0000 UTC m=+8.690500794" watchObservedRunningTime="2026-10-30 18:36:06.088254524 +0000 UTC m=+9.268276648"
Oct 30 18:36:06.391522 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:06.391489    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vtp2j" event={"ID":"b9849d01-613f-439d-86de-b517ee4a3341","Type":"ContainerStarted","Data":"b7def92fede361aea5c88894fc6124d74d5dc5a7d326479c9b3e5cdc15c81d3c"}
Oct 30 18:36:07.060806 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:07.060772    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92f38b64-26fb-42d6-826a-94cd37e8b668-original-pull-secret\") pod \"global-pull-secret-syncer-vs8lx\" (UID: \"92f38b64-26fb-42d6-826a-94cd37e8b668\") " pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:07.060987 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:07.060825    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs\") pod \"network-metrics-daemon-sqkfq\" (UID: \"6df0f1e7-07b7-4231-bc70-ba81aef29856\") " pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:07.060987 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:07.060858    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb856\" (UniqueName: \"kubernetes.io/projected/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278-kube-api-access-vb856\") pod \"network-check-target-xcmps\" (UID: \"0b0aa783-d4ab-490c-a5f7-d7da5ad8d278\") " pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:36:07.061131 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:07.061007    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:36:07.061131 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:07.061026    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:36:07.061131 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:07.061039    2572 projected.go:196] Error preparing data for projected volume kube-api-access-vb856 for pod openshift-network-diagnostics/network-check-target-xcmps: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:07.061131 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:07.061110    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278-kube-api-access-vb856 podName:0b0aa783-d4ab-490c-a5f7-d7da5ad8d278 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:15.061092843 +0000 UTC m=+18.241114956 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vb856" (UniqueName: "kubernetes.io/projected/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278-kube-api-access-vb856") pod "network-check-target-xcmps" (UID: "0b0aa783-d4ab-490c-a5f7-d7da5ad8d278") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:07.061539 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:07.061520    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:07.061611 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:07.061577    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92f38b64-26fb-42d6-826a-94cd37e8b668-original-pull-secret podName:92f38b64-26fb-42d6-826a-94cd37e8b668 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:15.061561656 +0000 UTC m=+18.241583761 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92f38b64-26fb-42d6-826a-94cd37e8b668-original-pull-secret") pod "global-pull-secret-syncer-vs8lx" (UID: "92f38b64-26fb-42d6-826a-94cd37e8b668") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:07.061676 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:07.061664    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:07.061715 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:07.061703    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs podName:6df0f1e7-07b7-4231-bc70-ba81aef29856 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:15.061692048 +0000 UTC m=+18.241714152 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs") pod "network-metrics-daemon-sqkfq" (UID: "6df0f1e7-07b7-4231-bc70-ba81aef29856") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:07.312686 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:07.312612    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:07.312830 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:07.312743    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqkfq" podUID="6df0f1e7-07b7-4231-bc70-ba81aef29856"
Oct 30 18:36:07.313208 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:07.313144    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:36:07.313306 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:07.313232    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xcmps" podUID="0b0aa783-d4ab-490c-a5f7-d7da5ad8d278"
Oct 30 18:36:07.313306 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:07.313260    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:07.313407 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:07.313313    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8lx" podUID="92f38b64-26fb-42d6-826a-94cd37e8b668"
Oct 30 18:36:07.398489 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:07.398458    2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:36:07.399499 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:07.398998    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz" event={"ID":"9092fa02-cc22-41a2-b4ee-070298ecab99","Type":"ContainerStarted","Data":"ac50c9ae5ccafa65873a9134c912cc9098c2c5b4984f5a95e2daeaa4b2f2b51c"}
Oct 30 18:36:08.398484 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:08.398373    2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 18:36:08.402577 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:08.402501    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz" event={"ID":"9092fa02-cc22-41a2-b4ee-070298ecab99","Type":"ContainerStarted","Data":"485b7ce2d247ca95cc62bd23b12eea115014f9c16285e0a6ef77fd876091dd27"}
Oct 30 18:36:09.256329 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:09.256178    2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:36:08.398391305Z","UUID":"675ccc6c-6de0-4cac-9303-db0040b7212b","Handler":null,"Name":"","Endpoint":""}
Oct 30 18:36:09.257780 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:09.257748    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 18:36:09.257780 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:09.257777    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 18:36:09.310100 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:09.307333    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:09.310100 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:09.307471    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqkfq" podUID="6df0f1e7-07b7-4231-bc70-ba81aef29856"
Oct 30 18:36:09.310100 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:09.307530    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:36:09.310100 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:09.307598    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xcmps" podUID="0b0aa783-d4ab-490c-a5f7-d7da5ad8d278"
Oct 30 18:36:09.310100 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:09.307639    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:09.310100 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:09.307698    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8lx" podUID="92f38b64-26fb-42d6-826a-94cd37e8b668"
Oct 30 18:36:09.407198 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:09.407161    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wd7km" event={"ID":"74a85732-6555-48bf-9ece-4e863cd1fb93","Type":"ContainerStarted","Data":"19d4e3efff67ab281119dd4acdd609cce95d548a9c83bebdcdf662321e6eeab0"}
Oct 30 18:36:09.413542 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:09.411326    2572 generic.go:356] "Generic (PLEG): container finished" podID="65beb5a5-1df3-42c8-9e14-769b4ad47255" containerID="cc8930523543e9e5f841136ae664aa02bf6216f23f6cca9c4a6fdb9a7b6fc259" exitCode=0
Oct 30 18:36:09.413542 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:09.411453    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h44bp" event={"ID":"65beb5a5-1df3-42c8-9e14-769b4ad47255","Type":"ContainerDied","Data":"cc8930523543e9e5f841136ae664aa02bf6216f23f6cca9c4a6fdb9a7b6fc259"}
Oct 30 18:36:09.414421 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:09.414244    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cdf94" event={"ID":"90df09a3-8706-4883-b711-be63e8e9f191","Type":"ContainerStarted","Data":"59d3ecc1acfd003baecfb2f5794a3852d72bc5d5ae267798e62365b533058ae3"}
Oct 30 18:36:09.526764 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:09.526668    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vtp2j" podStartSLOduration=5.517652114 podStartE2EDuration="11.526650803s" totalImagesPullingTime="6.008998689s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:06.506264413 +0000 UTC m=+9.686286536" watchObservedRunningTime="2026-10-30 18:36:09.526650803 +0000 UTC m=+12.706672926"
Oct 30 18:36:09.527448 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:09.527407    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-wd7km" podStartSLOduration=2.7725725260000003 podStartE2EDuration="11.527397144s" totalImagesPullingTime="8.754824618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:09.526496169 +0000 UTC m=+12.706518298" watchObservedRunningTime="2026-10-30 18:36:09.527397144 +0000 UTC m=+12.707419266"
Oct 30 18:36:09.641988 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:09.641932    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-cdf94" podStartSLOduration=2.593629166 podStartE2EDuration="11.641914419s" totalImagesPullingTime="9.048285253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:09.641571268 +0000 UTC m=+12.821593388" watchObservedRunningTime="2026-10-30 18:36:09.641914419 +0000 UTC m=+12.821936541"
Oct 30 18:36:10.418443 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:10.418054    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-44cfg" event={"ID":"f416f20b-a466-4f3d-bd2e-bb825b189067","Type":"ContainerStarted","Data":"10cc213b9bd4949816d459a7a9afac9fa4a1c59c29c806cca88e3ddf9dceddce"}
Oct 30 18:36:11.310714 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:11.310688    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:11.310714 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:11.310707    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:36:11.310881 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:11.310728    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:11.310881 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:11.310810    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqkfq" podUID="6df0f1e7-07b7-4231-bc70-ba81aef29856"
Oct 30 18:36:11.310954 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:11.310914    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xcmps" podUID="0b0aa783-d4ab-490c-a5f7-d7da5ad8d278"
Oct 30 18:36:11.311004 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:11.310984    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8lx" podUID="92f38b64-26fb-42d6-826a-94cd37e8b668"
Oct 30 18:36:11.422876 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:11.422840    2572 generic.go:356] "Generic (PLEG): container finished" podID="65beb5a5-1df3-42c8-9e14-769b4ad47255" containerID="55d55076763e6a9a0ce5929024b3c0027fe2703b8d66edf482263c715dd6e4b8" exitCode=0
Oct 30 18:36:11.423470 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:11.422944    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h44bp" event={"ID":"65beb5a5-1df3-42c8-9e14-769b4ad47255","Type":"ContainerDied","Data":"55d55076763e6a9a0ce5929024b3c0027fe2703b8d66edf482263c715dd6e4b8"}
Oct 30 18:36:11.424636 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:11.424591    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bhxxm" event={"ID":"3935b0fd-69be-4272-adc5-ecfbc7af04ae","Type":"ContainerStarted","Data":"8467f676f494ee00b3a048e49e2c90d0a5df5ba7a5fb5e006072bd44b54590d7"}
Oct 30 18:36:11.426969 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:11.426939    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz" event={"ID":"9092fa02-cc22-41a2-b4ee-070298ecab99","Type":"ContainerStarted","Data":"b8c535396765e51d6a7f344f0922967aba784d571312b607df13059cdaed4433"}
Oct 30 18:36:11.641078 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:11.641000    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-44cfg" podStartSLOduration=4.968845468 podStartE2EDuration="13.640984819s" totalImagesPullingTime="8.672139351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:10.532469279 +0000 UTC m=+13.712491399" watchObservedRunningTime="2026-10-30 18:36:11.640984819 +0000 UTC m=+14.821006942"
Oct 30 18:36:11.759747 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:11.759694    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz" podStartSLOduration=3.459429767 podStartE2EDuration="13.759679502s" totalImagesPullingTime="10.300249735s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:11.75930377 +0000 UTC m=+14.939325891" watchObservedRunningTime="2026-10-30 18:36:11.759679502 +0000 UTC m=+14.939701619"
Oct 30 18:36:11.880330 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:11.880276    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bhxxm" podStartSLOduration=2.7295131059999997 podStartE2EDuration="13.880257478s" totalImagesPullingTime="11.150744372s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:11.880060169 +0000 UTC m=+15.060082292" watchObservedRunningTime="2026-10-30 18:36:11.880257478 +0000 UTC m=+15.060279600"
Oct 30 18:36:13.307048 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:13.307018    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:13.307628 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:13.307086    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:36:13.307628 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:13.307093    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:13.307628 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:13.307175    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8lx" podUID="92f38b64-26fb-42d6-826a-94cd37e8b668"
Oct 30 18:36:13.307628 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:13.307285    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xcmps" podUID="0b0aa783-d4ab-490c-a5f7-d7da5ad8d278"
Oct 30 18:36:13.307628 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:13.307370    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqkfq" podUID="6df0f1e7-07b7-4231-bc70-ba81aef29856"
Oct 30 18:36:13.434361 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:13.434331    2572 generic.go:356] "Generic (PLEG): container finished" podID="65beb5a5-1df3-42c8-9e14-769b4ad47255" containerID="ed3cf5ccf33456eff4cf9ddf778547bb8117b02ebfa3e208fd1c2c507e6244d3" exitCode=0
Oct 30 18:36:13.434508 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:13.434391    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h44bp" event={"ID":"65beb5a5-1df3-42c8-9e14-769b4ad47255","Type":"ContainerDied","Data":"ed3cf5ccf33456eff4cf9ddf778547bb8117b02ebfa3e208fd1c2c507e6244d3"}
Oct 30 18:36:15.122992 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:15.122845    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92f38b64-26fb-42d6-826a-94cd37e8b668-original-pull-secret\") pod \"global-pull-secret-syncer-vs8lx\" (UID: \"92f38b64-26fb-42d6-826a-94cd37e8b668\") " pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:15.122992 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:15.122981    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:15.123469 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:15.123012    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs\") pod \"network-metrics-daemon-sqkfq\" (UID: \"6df0f1e7-07b7-4231-bc70-ba81aef29856\") " pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:15.123469 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:15.123028    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92f38b64-26fb-42d6-826a-94cd37e8b668-original-pull-secret podName:92f38b64-26fb-42d6-826a-94cd37e8b668 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:31.12301494 +0000 UTC m=+34.303037040 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/92f38b64-26fb-42d6-826a-94cd37e8b668-original-pull-secret") pod "global-pull-secret-syncer-vs8lx" (UID: "92f38b64-26fb-42d6-826a-94cd37e8b668") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:15.123469 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:15.123060    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb856\" (UniqueName: \"kubernetes.io/projected/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278-kube-api-access-vb856\") pod \"network-check-target-xcmps\" (UID: \"0b0aa783-d4ab-490c-a5f7-d7da5ad8d278\") " pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:36:15.123469 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:15.123123    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:15.123469 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:15.123172    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:36:15.123469 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:15.123177    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs podName:6df0f1e7-07b7-4231-bc70-ba81aef29856 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:31.123162861 +0000 UTC m=+34.303184973 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs") pod "network-metrics-daemon-sqkfq" (UID: "6df0f1e7-07b7-4231-bc70-ba81aef29856") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:15.123469 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:15.123181    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:36:15.123469 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:15.123190    2572 projected.go:196] Error preparing data for projected volume kube-api-access-vb856 for pod openshift-network-diagnostics/network-check-target-xcmps: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:15.123469 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:15.123217    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278-kube-api-access-vb856 podName:0b0aa783-d4ab-490c-a5f7-d7da5ad8d278 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:31.123209753 +0000 UTC m=+34.303231853 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vb856" (UniqueName: "kubernetes.io/projected/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278-kube-api-access-vb856") pod "network-check-target-xcmps" (UID: "0b0aa783-d4ab-490c-a5f7-d7da5ad8d278") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:15.306642 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:15.306615    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:15.306810 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:15.306669    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:36:15.306810 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:15.306678    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:15.306810 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:15.306760    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8lx" podUID="92f38b64-26fb-42d6-826a-94cd37e8b668"
Oct 30 18:36:15.306962 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:15.306852    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xcmps" podUID="0b0aa783-d4ab-490c-a5f7-d7da5ad8d278"
Oct 30 18:36:15.306962 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:15.306948    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqkfq" podUID="6df0f1e7-07b7-4231-bc70-ba81aef29856"
Oct 30 18:36:15.443443 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:15.443367    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r" event={"ID":"e1c10266-f60d-4b43-b2ee-cef69669bc7b","Type":"ContainerStarted","Data":"021ecb95ad65e405076b5f7a602ef74ddf0e7eff2753cc6a8305491d2003c9c1"}
Oct 30 18:36:15.443591 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:15.443451    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r" event={"ID":"e1c10266-f60d-4b43-b2ee-cef69669bc7b","Type":"ContainerStarted","Data":"d4bf6b8fba421122af58b054671312a549c547f9f445181f829d1a2baa20c637"}
Oct 30 18:36:15.443591 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:15.443487    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r" event={"ID":"e1c10266-f60d-4b43-b2ee-cef69669bc7b","Type":"ContainerStarted","Data":"3d97ae34a894992cce39b1c3eb96d5b3fba56e173d60d757a79ebb240c718731"}
Oct 30 18:36:15.443591 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:15.443516    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r" event={"ID":"e1c10266-f60d-4b43-b2ee-cef69669bc7b","Type":"ContainerStarted","Data":"17623d095f8cf6c5b3abf33c95404f94faf254186cdbffd7bb421e4cdfe9026f"}
Oct 30 18:36:15.443591 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:15.443551    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r" event={"ID":"e1c10266-f60d-4b43-b2ee-cef69669bc7b","Type":"ContainerStarted","Data":"df792e4be89eacdafab39d4c911aa5986f37fd8fc331b98ece886303f931e22b"}
Oct 30 18:36:15.443591 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:15.443589    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r" event={"ID":"e1c10266-f60d-4b43-b2ee-cef69669bc7b","Type":"ContainerStarted","Data":"4cb2d9b39b9226b180dab442c7ab74602a23e27151f82af5d9c59a31b09f365a"}
Oct 30 18:36:16.448196 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:16.448163    2572 generic.go:356] "Generic (PLEG): container finished" podID="65beb5a5-1df3-42c8-9e14-769b4ad47255" containerID="1b14c6301e6f00ba59027cfce1260b9e8140986d05171a6dcb81c3ced71dfeb6" exitCode=0
Oct 30 18:36:16.448196 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:16.448201    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h44bp" event={"ID":"65beb5a5-1df3-42c8-9e14-769b4ad47255","Type":"ContainerDied","Data":"1b14c6301e6f00ba59027cfce1260b9e8140986d05171a6dcb81c3ced71dfeb6"}
Oct 30 18:36:17.309914 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:17.309885    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:17.310005 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:17.309965    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:17.310005 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:17.309991    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:36:17.310101 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:17.310058    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8lx" podUID="92f38b64-26fb-42d6-826a-94cd37e8b668"
Oct 30 18:36:17.310155 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:17.310135    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xcmps" podUID="0b0aa783-d4ab-490c-a5f7-d7da5ad8d278"
Oct 30 18:36:17.310244 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:17.310224    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqkfq" podUID="6df0f1e7-07b7-4231-bc70-ba81aef29856"
Oct 30 18:36:17.453343 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:17.453314    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r" event={"ID":"e1c10266-f60d-4b43-b2ee-cef69669bc7b","Type":"ContainerStarted","Data":"b60698b123db0a0f6cac1edc3ef3632b9514935bf73bd84c8ac90c001d41a12d"}
Oct 30 18:36:17.456223 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:17.456200    2572 generic.go:356] "Generic (PLEG): container finished" podID="65beb5a5-1df3-42c8-9e14-769b4ad47255" containerID="af02f46d8eb4a9d0283a67ee6f6c588f2d9cb834b565def1ea574e349a30a56e" exitCode=0
Oct 30 18:36:17.456354 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:17.456234    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h44bp" event={"ID":"65beb5a5-1df3-42c8-9e14-769b4ad47255","Type":"ContainerDied","Data":"af02f46d8eb4a9d0283a67ee6f6c588f2d9cb834b565def1ea574e349a30a56e"}
Oct 30 18:36:18.461154 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:18.461117    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h44bp" event={"ID":"65beb5a5-1df3-42c8-9e14-769b4ad47255","Type":"ContainerStarted","Data":"6e53a4015847f4ccf894961fac0cca9263bbaeeb44523d810fe65e303c3b124a"}
Oct 30 18:36:18.685839 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:18.685791    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-h44bp" podStartSLOduration=7.723572449 podStartE2EDuration="20.685779055s" totalImagesPullingTime="12.962206606s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:18.685375726 +0000 UTC m=+21.865397849" watchObservedRunningTime="2026-10-30 18:36:18.685779055 +0000 UTC m=+21.865801192"
Oct 30 18:36:19.154280 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:19.154237    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5v2r9"
Oct 30 18:36:19.154449 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:19.154410    2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:36:19.154806 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:19.154792    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5v2r9"
Oct 30 18:36:19.309880 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:19.309738    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:19.309880 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:19.309756    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:36:19.309880 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:19.309850    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqkfq" podUID="6df0f1e7-07b7-4231-bc70-ba81aef29856"
Oct 30 18:36:19.309880 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:19.309877    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:19.310101 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:19.309937    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xcmps" podUID="0b0aa783-d4ab-490c-a5f7-d7da5ad8d278"
Oct 30 18:36:19.310101 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:19.310016    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8lx" podUID="92f38b64-26fb-42d6-826a-94cd37e8b668"
Oct 30 18:36:19.469630 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:19.469146    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r" event={"ID":"e1c10266-f60d-4b43-b2ee-cef69669bc7b","Type":"ContainerStarted","Data":"bd011e9a4d51902dc6e3fd841a05f08069989154350e57e0089bf7d62a299300"}
Oct 30 18:36:19.469630 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:19.469614    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:19.470587 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:19.469687    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:19.470587 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:19.469735    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:19.485462 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:19.485419    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:19.486239 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:19.486224    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:19.674836 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:19.674640    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r" podStartSLOduration=6.403035352 podStartE2EDuration="21.674626887s" totalImagesPullingTime="15.271591535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:19.674292732 +0000 UTC m=+22.854314852" watchObservedRunningTime="2026-10-30 18:36:19.674626887 +0000 UTC m=+22.854649007"
Oct 30 18:36:21.307023 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:21.306992    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:21.307023 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:21.307023    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:21.307512 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:21.307105    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqkfq" podUID="6df0f1e7-07b7-4231-bc70-ba81aef29856"
Oct 30 18:36:21.307512 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:21.307203    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8lx" podUID="92f38b64-26fb-42d6-826a-94cd37e8b668"
Oct 30 18:36:21.307512 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:21.307260    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:36:21.307512 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:21.307313    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xcmps" podUID="0b0aa783-d4ab-490c-a5f7-d7da5ad8d278"
Oct 30 18:36:21.972934 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:21.972895    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vs8lx"]
Oct 30 18:36:21.973087 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:21.973006    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:21.973127 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:21.973113    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8lx" podUID="92f38b64-26fb-42d6-826a-94cd37e8b668"
Oct 30 18:36:21.975611 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:21.975582    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xcmps"]
Oct 30 18:36:21.975707 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:21.975659    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:36:21.975758 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:21.975725    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xcmps" podUID="0b0aa783-d4ab-490c-a5f7-d7da5ad8d278"
Oct 30 18:36:22.023146 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:22.023118    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sqkfq"]
Oct 30 18:36:22.023247 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:22.023214    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:22.023336 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:22.023317    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqkfq" podUID="6df0f1e7-07b7-4231-bc70-ba81aef29856"
Oct 30 18:36:23.307462 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:23.307433    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:23.307899 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:23.307444    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:23.307899 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:23.307543    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8lx" podUID="92f38b64-26fb-42d6-826a-94cd37e8b668"
Oct 30 18:36:23.307899 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:23.307656    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqkfq" podUID="6df0f1e7-07b7-4231-bc70-ba81aef29856"
Oct 30 18:36:24.307515 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:24.307436    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:36:24.307935 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:24.307537    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xcmps" podUID="0b0aa783-d4ab-490c-a5f7-d7da5ad8d278"
Oct 30 18:36:25.306611 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:25.306581    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:25.306755 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:25.306706    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vs8lx" podUID="92f38b64-26fb-42d6-826a-94cd37e8b668"
Oct 30 18:36:25.306829 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:25.306750    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:25.306878 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:25.306842    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sqkfq" podUID="6df0f1e7-07b7-4231-bc70-ba81aef29856"
Oct 30 18:36:26.307490 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:26.307407    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:36:26.307934 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:26.307505    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xcmps" podUID="0b0aa783-d4ab-490c-a5f7-d7da5ad8d278"
Oct 30 18:36:27.076178 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.076153    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeReady"
Oct 30 18:36:27.076342 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.076283    2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 18:36:27.189668 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.189594    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-f4bz7"]
Oct 30 18:36:27.193353 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.193333    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wl4tz"]
Oct 30 18:36:27.193482 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.193453    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-f4bz7"
Oct 30 18:36:27.196888 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.196871    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-65rz2"]
Oct 30 18:36:27.196985 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.196975    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wl4tz"
Oct 30 18:36:27.200456 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.200436    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-65rz2"
Oct 30 18:36:27.240143 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.240121    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8598c87c5b-d4m59"]
Oct 30 18:36:27.243421 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.243406    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b777d677-gcvwt"]
Oct 30 18:36:27.243576 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.243519    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:27.247197 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.247180    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"]
Oct 30 18:36:27.247301 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.247287    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:27.250450 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.250431    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rpvxs"]
Oct 30 18:36:27.250554 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.250540    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:27.253720 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.253703    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-776vb"]
Oct 30 18:36:27.253806 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.253792    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rpvxs"
Oct 30 18:36:27.256783 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.256765    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"]
Oct 30 18:36:27.256913 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.256885    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:27.259890 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.259874    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"]
Oct 30 18:36:27.259979 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.259934    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"
Oct 30 18:36:27.262889 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.262873    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-b44dd54dc-hqb7p"]
Oct 30 18:36:27.263001 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.262985    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:27.265814 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.265798    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-wpnq5"]
Oct 30 18:36:27.265902 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.265854    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:27.269267 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.269250    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wpnq5"
Oct 30 18:36:27.290505 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.290473    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 18:36:27.290604 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.290569    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 18:36:27.291112 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.291093    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk"]
Oct 30 18:36:27.299657 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.299634    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 18:36:27.299761 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.299735    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 18:36:27.299761 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.299751    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 18:36:27.300496 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.300053    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-h758x\""
Oct 30 18:36:27.303915 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.303864    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6pktt"]
Oct 30 18:36:27.304220 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.304191    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk"
Oct 30 18:36:27.306960 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.306871    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fswdh"]
Oct 30 18:36:27.306960 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.306933    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6pktt"
Oct 30 18:36:27.307149 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.306986    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:27.307428 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.307412    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 18:36:27.307801 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.307782    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:27.309993 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.309976    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sxk64"]
Oct 30 18:36:27.310133 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.310117    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fswdh"
Oct 30 18:36:27.312924 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.312907    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-g6sxb"]
Oct 30 18:36:27.313046 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.313029    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:27.313261 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.313242    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dh8q\" (UniqueName: \"kubernetes.io/projected/a20a00eb-cde2-4df4-a191-0502fcaf288d-kube-api-access-5dh8q\") pod \"kas-connection-checker-685895c7c9-65rz2\" (UID: \"a20a00eb-cde2-4df4-a191-0502fcaf288d\") " pod="kube-system/kas-connection-checker-685895c7c9-65rz2"
Oct 30 18:36:27.313344 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.313277    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/92981ab7-a41b-482f-98c3-a328e5e8d6f0-ca-trust-extracted\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:27.313420 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.313392    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-bound-sa-token\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:27.313492 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.313476    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:27.313525 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.313508    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92981ab7-a41b-482f-98c3-a328e5e8d6f0-trusted-ca\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:27.313573 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.313534    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be409e3c-3420-4250-bfa7-3b01ecd69d9b-image-registry-private-configuration\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:27.313573 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.313562    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be409e3c-3420-4250-bfa7-3b01ecd69d9b-installation-pull-secrets\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:27.313645 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.313595    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-bound-sa-token\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:27.313645 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.313637    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-certificates\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:27.313716 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.313653    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dth6b\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-kube-api-access-dth6b\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:27.313716 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.313668    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdblb\" (UniqueName: \"kubernetes.io/projected/80e45018-1682-47cd-9523-dbeaf35a39cd-kube-api-access-vdblb\") pod \"cluster-monitoring-operator-884576c57-plkxx\" (UID: \"80e45018-1682-47cd-9523-dbeaf35a39cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:27.313716 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.313694    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:27.313826 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.313763    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hq7p\" (UniqueName: \"kubernetes.io/projected/e438abba-e5b6-41bb-bc67-1a1d322e6534-kube-api-access-6hq7p\") pod \"kas-connection-checker-685895c7c9-f4bz7\" (UID: \"e438abba-e5b6-41bb-bc67-1a1d322e6534\") " pod="kube-system/kas-connection-checker-685895c7c9-f4bz7"
Oct 30 18:36:27.313826 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.313783    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hp67\" (UniqueName: \"kubernetes.io/projected/d8842256-47fc-4dc5-91f6-65938f410080-kube-api-access-5hp67\") pod \"kas-connection-checker-685895c7c9-wl4tz\" (UID: \"d8842256-47fc-4dc5-91f6-65938f410080\") " pod="kube-system/kas-connection-checker-685895c7c9-wl4tz"
Oct 30 18:36:27.313826 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.313803    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/92981ab7-a41b-482f-98c3-a328e5e8d6f0-image-registry-private-configuration\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:27.313913 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.313848    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be409e3c-3420-4250-bfa7-3b01ecd69d9b-ca-trust-extracted\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:27.313913 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.313876    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-certificates\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:27.313913 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.313902    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be409e3c-3420-4250-bfa7-3b01ecd69d9b-trusted-ca\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:27.313998 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.313967    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/92981ab7-a41b-482f-98c3-a328e5e8d6f0-installation-pull-secrets\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:27.314046 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.314030    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xg8f\" (UniqueName: \"kubernetes.io/projected/5d05b776-9750-4c5b-8d91-4cf27329a1fc-kube-api-access-2xg8f\") pod \"volume-data-source-validator-58c9f5f46d-rpvxs\" (UID: \"5d05b776-9750-4c5b-8d91-4cf27329a1fc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rpvxs"
Oct 30 18:36:27.314104 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.314078    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knktl\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-kube-api-access-knktl\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:27.314104 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.314100    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-plkxx\" (UID: \"80e45018-1682-47cd-9523-dbeaf35a39cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:27.314242 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.314138    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/80e45018-1682-47cd-9523-dbeaf35a39cd-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-plkxx\" (UID: \"80e45018-1682-47cd-9523-dbeaf35a39cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:27.315748 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.315734    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g6sxb"
Oct 30 18:36:27.319294 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.319277    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-65rz2"]
Oct 30 18:36:27.319381 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.319317    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-f4bz7"]
Oct 30 18:36:27.319381 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.319339    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8598c87c5b-d4m59"]
Oct 30 18:36:27.319381 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.319361    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"]
Oct 30 18:36:27.319381 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.319380    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rpvxs"]
Oct 30 18:36:27.319535 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.319399    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"]
Oct 30 18:36:27.319535 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.319418    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"]
Oct 30 18:36:27.319535 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.319454    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-b44dd54dc-hqb7p"]
Oct 30 18:36:27.341555 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.341531    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk"]
Oct 30 18:36:27.345129 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.345108    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wl4tz"]
Oct 30 18:36:27.347165 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.347145    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6pktt"]
Oct 30 18:36:27.347290 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.347186    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b777d677-gcvwt"]
Oct 30 18:36:27.348230 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.348204    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 18:36:27.348700 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.348680    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:27.349047 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.348822    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 18:36:27.349047 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.348913    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 18:36:27.349047 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.348936    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 18:36:27.349247 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.349195    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 18:36:27.349247 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.349216    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 18:36:27.349247 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.349226    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:27.349247 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.349195    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:27.349391 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.349258    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 18:36:27.349391 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.349229    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 18:36:27.349391 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.349244    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:27.349584 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.349571    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 18:36:27.349785 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.349766    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:27.349864 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.349776    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:27.349864 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.349835    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:27.349936 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.349893    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 18:36:27.349982 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.349936    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:27.350032 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.350003    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 18:36:27.405213 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.405185    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 18:36:27.409785 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.409762    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-776vb"]
Oct 30 18:36:27.414377 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.414358    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hp67\" (UniqueName: \"kubernetes.io/projected/d8842256-47fc-4dc5-91f6-65938f410080-kube-api-access-5hp67\") pod \"kas-connection-checker-685895c7c9-wl4tz\" (UID: \"d8842256-47fc-4dc5-91f6-65938f410080\") " pod="kube-system/kas-connection-checker-685895c7c9-wl4tz"
Oct 30 18:36:27.414448 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.414387    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-default-certificate\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:27.414448 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.414405    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:27.414448 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.414420    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmg9f\" (UniqueName: \"kubernetes.io/projected/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-kube-api-access-dmg9f\") pod \"cluster-samples-operator-6ddb6c8b94-2xb8v\" (UID: \"b94a90ae-e06a-4e60-a48e-b567ce6e4edd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"
Oct 30 18:36:27.414550 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.414486    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmjgg\" (UniqueName: \"kubernetes.io/projected/05fe2f04-29f7-4999-87ef-b30035b34883-kube-api-access-mmjgg\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:27.414550 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.414509    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12928ad7-4038-473a-a4aa-be1a6a42eabc-tmp-dir\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:27.414550 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.414529    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0648f559-00ac-4de0-8b9f-ce02d1b657a9-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk\" (UID: \"0648f559-00ac-4de0-8b9f-ce02d1b657a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk"
Oct 30 18:36:27.414639 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.414606    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-certificates\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:27.414675 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.414651    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e954d56b-ace6-4539-a467-486c1e889886-config\") pod \"service-ca-operator-544d749c8c-wpnq5\" (UID: \"e954d56b-ace6-4539-a467-486c1e889886\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wpnq5"
Oct 30 18:36:27.414714 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.414689    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-service-ca-bundle\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:27.414776 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.414763    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcz45\" (UniqueName: \"kubernetes.io/projected/12928ad7-4038-473a-a4aa-be1a6a42eabc-kube-api-access-qcz45\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:27.414833 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.414821    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be409e3c-3420-4250-bfa7-3b01ecd69d9b-trusted-ca\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:27.414873 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.414841    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-stats-auth\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:27.414873 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.414857    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9b63e008-3071-4593-bba8-3bcc673b5dc2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6pktt\" (UID: \"9b63e008-3071-4593-bba8-3bcc673b5dc2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6pktt"
Oct 30 18:36:27.414969 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.414958    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/92981ab7-a41b-482f-98c3-a328e5e8d6f0-installation-pull-secrets\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:27.415005 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.414980    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21b6f672-2dc5-4523-9dfa-6dfaa30a8bac-cert\") pod \"ingress-canary-g6sxb\" (UID: \"21b6f672-2dc5-4523-9dfa-6dfaa30a8bac\") " pod="openshift-ingress-canary/ingress-canary-g6sxb"
Oct 30 18:36:27.415005 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.414998    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/80e45018-1682-47cd-9523-dbeaf35a39cd-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-plkxx\" (UID: \"80e45018-1682-47cd-9523-dbeaf35a39cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:27.415107 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.415015    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-snapshots\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:27.415147 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.415118    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dh8q\" (UniqueName: \"kubernetes.io/projected/a20a00eb-cde2-4df4-a191-0502fcaf288d-kube-api-access-5dh8q\") pod \"kas-connection-checker-685895c7c9-65rz2\" (UID: \"a20a00eb-cde2-4df4-a191-0502fcaf288d\") " pod="kube-system/kas-connection-checker-685895c7c9-65rz2"
Oct 30 18:36:27.415219 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.415201    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e954d56b-ace6-4539-a467-486c1e889886-serving-cert\") pod \"service-ca-operator-544d749c8c-wpnq5\" (UID: \"e954d56b-ace6-4539-a467-486c1e889886\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wpnq5"
Oct 30 18:36:27.415263 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.415239    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks7sd\" (UniqueName: \"kubernetes.io/projected/21b6f672-2dc5-4523-9dfa-6dfaa30a8bac-kube-api-access-ks7sd\") pod \"ingress-canary-g6sxb\" (UID: \"21b6f672-2dc5-4523-9dfa-6dfaa30a8bac\") " pod="openshift-ingress-canary/ingress-canary-g6sxb"
Oct 30 18:36:27.415314 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.415267    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-serving-cert\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:27.415314 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.415293    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-certificates\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:27.415417 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.415318    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:27.415417 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.415370    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0648f559-00ac-4de0-8b9f-ce02d1b657a9-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk\" (UID: \"0648f559-00ac-4de0-8b9f-ce02d1b657a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk"
Oct 30 18:36:27.415490 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.415413    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-tmp\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:27.415490 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.415467    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-plkxx\" (UID: \"80e45018-1682-47cd-9523-dbeaf35a39cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:27.415589 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.415520    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2xb8v\" (UID: \"b94a90ae-e06a-4e60-a48e-b567ce6e4edd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"
Oct 30 18:36:27.415589 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:27.415538    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:27.415589 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.415552    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be409e3c-3420-4250-bfa7-3b01ecd69d9b-image-registry-private-configuration\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:27.415723 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:27.415593    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls podName:80e45018-1682-47cd-9523-dbeaf35a39cd nodeName:}" failed. No retries permitted until 2026-10-30 18:36:27.915575734 +0000 UTC m=+31.095597868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-plkxx" (UID: "80e45018-1682-47cd-9523-dbeaf35a39cd") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:27.415723 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.415625    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12928ad7-4038-473a-a4aa-be1a6a42eabc-config-volume\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:27.415723 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.415663    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-bound-sa-token\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:27.415723 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.415706    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60e0e397-bb2c-4637-9376-c5af871666cb-trusted-ca\") pod \"console-operator-7cd56b8dbc-jkgcq\" (UID: \"60e0e397-bb2c-4637-9376-c5af871666cb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:27.415913 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.415744    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8ggz\" (UniqueName: \"kubernetes.io/projected/0648f559-00ac-4de0-8b9f-ce02d1b657a9-kube-api-access-k8ggz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk\" (UID: \"0648f559-00ac-4de0-8b9f-ce02d1b657a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk"
Oct 30 18:36:27.415913 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.415789    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92981ab7-a41b-482f-98c3-a328e5e8d6f0-trusted-ca\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:27.415913 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.415818    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdblb\" (UniqueName: \"kubernetes.io/projected/80e45018-1682-47cd-9523-dbeaf35a39cd-kube-api-access-vdblb\") pod \"cluster-monitoring-operator-884576c57-plkxx\" (UID: \"80e45018-1682-47cd-9523-dbeaf35a39cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:27.415913 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.415853    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/80e45018-1682-47cd-9523-dbeaf35a39cd-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-plkxx\" (UID: \"80e45018-1682-47cd-9523-dbeaf35a39cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:27.415913 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.415864    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hq7p\" (UniqueName: \"kubernetes.io/projected/e438abba-e5b6-41bb-bc67-1a1d322e6534-kube-api-access-6hq7p\") pod \"kas-connection-checker-685895c7c9-f4bz7\" (UID: \"e438abba-e5b6-41bb-bc67-1a1d322e6534\") " pod="kube-system/kas-connection-checker-685895c7c9-f4bz7"
Oct 30 18:36:27.415913 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.415900    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60e0e397-bb2c-4637-9376-c5af871666cb-serving-cert\") pod \"console-operator-7cd56b8dbc-jkgcq\" (UID: \"60e0e397-bb2c-4637-9376-c5af871666cb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:27.416205 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.415931    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:27.416205 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.415961    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/92981ab7-a41b-482f-98c3-a328e5e8d6f0-image-registry-private-configuration\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:27.416205 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.415998    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be409e3c-3420-4250-bfa7-3b01ecd69d9b-ca-trust-extracted\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:27.416205 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:27.416046    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:27.416205 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:27.416075    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b777d677-gcvwt: secret "image-registry-tls" not found
Oct 30 18:36:27.416205 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:27.416124    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls podName:be409e3c-3420-4250-bfa7-3b01ecd69d9b nodeName:}" failed. No retries permitted until 2026-10-30 18:36:27.91610925 +0000 UTC m=+31.096131369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls") pod "image-registry-7b777d677-gcvwt" (UID: "be409e3c-3420-4250-bfa7-3b01ecd69d9b") : secret "image-registry-tls" not found
Oct 30 18:36:27.416205 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.416158    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/60e0e397-bb2c-4637-9376-c5af871666cb-config\") pod \"console-operator-7cd56b8dbc-jkgcq\" (UID: \"60e0e397-bb2c-4637-9376-c5af871666cb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:27.416205 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.416187    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6pktt\" (UID: \"9b63e008-3071-4593-bba8-3bcc673b5dc2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6pktt"
Oct 30 18:36:27.416571 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.416214    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qznzq\" (UniqueName: \"kubernetes.io/projected/e954d56b-ace6-4539-a467-486c1e889886-kube-api-access-qznzq\") pod \"service-ca-operator-544d749c8c-wpnq5\" (UID: \"e954d56b-ace6-4539-a467-486c1e889886\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wpnq5"
Oct 30 18:36:27.416571 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.416251    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xg8f\" (UniqueName: \"kubernetes.io/projected/5d05b776-9750-4c5b-8d91-4cf27329a1fc-kube-api-access-2xg8f\") pod \"volume-data-source-validator-58c9f5f46d-rpvxs\" (UID: \"5d05b776-9750-4c5b-8d91-4cf27329a1fc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rpvxs"
Oct 30 18:36:27.416571 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.416281    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-2xb8v\" (UID: \"b94a90ae-e06a-4e60-a48e-b567ce6e4edd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"
Oct 30 18:36:27.416571 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.416306    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx8hm\" (UniqueName: \"kubernetes.io/projected/5ef02004-71fc-4464-9550-bd094d27f21f-kube-api-access-dx8hm\") pod \"network-check-source-6f78d59f99-fswdh\" (UID: \"5ef02004-71fc-4464-9550-bd094d27f21f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fswdh"
Oct 30 18:36:27.416571 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.416337    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:27.416571 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.416404    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v526v\" (UniqueName: \"kubernetes.io/projected/60e0e397-bb2c-4637-9376-c5af871666cb-kube-api-access-v526v\") pod \"console-operator-7cd56b8dbc-jkgcq\" (UID: \"60e0e397-bb2c-4637-9376-c5af871666cb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:27.416571 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.416412    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be409e3c-3420-4250-bfa7-3b01ecd69d9b-ca-trust-extracted\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:27.416571 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.416566    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/92981ab7-a41b-482f-98c3-a328e5e8d6f0-ca-trust-extracted\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:27.416906 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.416600    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l74z8\" (UniqueName: \"kubernetes.io/projected/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-kube-api-access-l74z8\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:27.416906 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.416665    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:27.416906 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.416755    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-bound-sa-token\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:27.416906 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.416794    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knktl\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-kube-api-access-knktl\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:27.416906 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.416843    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:27.416906 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.416867    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be409e3c-3420-4250-bfa7-3b01ecd69d9b-installation-pull-secrets\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:27.416906 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.416890    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-certificates\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:27.417245 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.416911    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dth6b\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-kube-api-access-dth6b\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:27.417245 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:27.416929    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:27.417245 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:27.416945    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8598c87c5b-d4m59: secret "image-registry-tls" not found
Oct 30 18:36:27.417245 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:27.416988    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls podName:92981ab7-a41b-482f-98c3-a328e5e8d6f0 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:27.916972713 +0000 UTC m=+31.096994827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls") pod "image-registry-8598c87c5b-d4m59" (UID: "92981ab7-a41b-482f-98c3-a328e5e8d6f0") : secret "image-registry-tls" not found
Oct 30 18:36:27.417245 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.416866    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/92981ab7-a41b-482f-98c3-a328e5e8d6f0-ca-trust-extracted\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:27.417501 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.417479    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-certificates\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:27.419454 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.419294    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/92981ab7-a41b-482f-98c3-a328e5e8d6f0-image-registry-private-configuration\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:27.419537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.419414    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be409e3c-3420-4250-bfa7-3b01ecd69d9b-image-registry-private-configuration\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:27.419537 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.419412    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/92981ab7-a41b-482f-98c3-a328e5e8d6f0-installation-pull-secrets\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:27.419603 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.419589    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be409e3c-3420-4250-bfa7-3b01ecd69d9b-installation-pull-secrets\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:27.443948 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.443893    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 18:36:27.443948 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.443896    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 18:36:27.444296 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.444276    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 18:36:27.444366 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.444297    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 18:36:27.444425 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.444377    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 18:36:27.444425 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.444409    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 18:36:27.444760 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.444743    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 18:36:27.444831 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.444749    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 18:36:27.444831 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.444816    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 18:36:27.444940 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.444915    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 18:36:27.444940 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.444918    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 18:36:27.445082 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.445053    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vtx29\""
Oct 30 18:36:27.445082 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.445077    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cxkbs\""
Oct 30 18:36:27.445177 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.445089    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zmfwz\""
Oct 30 18:36:27.445177 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.445100    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-4cchw\""
Oct 30 18:36:27.445177 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.444938    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 18:36:27.445177 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.445107    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 18:36:27.445177 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.445126    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 18:36:27.445177 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.445132    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 18:36:27.445177 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.445168    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 18:36:27.445399 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.445181    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9ph7g\""
Oct 30 18:36:27.445399 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.445206    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sxk64"]
Oct 30 18:36:27.445795 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.445777    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be409e3c-3420-4250-bfa7-3b01ecd69d9b-trusted-ca\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:27.445886 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.445875    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-wpnq5"]
Oct 30 18:36:27.446613 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.446593    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92981ab7-a41b-482f-98c3-a328e5e8d6f0-trusted-ca\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:27.446812 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.446797    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g6sxb"]
Oct 30 18:36:27.447414 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.447399    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fswdh"]
Oct 30 18:36:27.448202 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.448189    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 18:36:27.448354 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.448338    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:27.448406 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.448362    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 18:36:27.448406 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.448397    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 18:36:27.448535 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.448517    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 18:36:27.449142 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.449130    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:27.449196 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.449139    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 18:36:27.449196 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.449152    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 18:36:27.449196 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.449176    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 18:36:27.474308 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.474262    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdblb\" (UniqueName: \"kubernetes.io/projected/80e45018-1682-47cd-9523-dbeaf35a39cd-kube-api-access-vdblb\") pod \"cluster-monitoring-operator-884576c57-plkxx\" (UID: \"80e45018-1682-47cd-9523-dbeaf35a39cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:27.474412 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.474388    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dh8q\" (UniqueName: \"kubernetes.io/projected/a20a00eb-cde2-4df4-a191-0502fcaf288d-kube-api-access-5dh8q\") pod \"kas-connection-checker-685895c7c9-65rz2\" (UID: \"a20a00eb-cde2-4df4-a191-0502fcaf288d\") " pod="kube-system/kas-connection-checker-685895c7c9-65rz2"
Oct 30 18:36:27.474637 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.474603    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-bound-sa-token\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:27.475053 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.475035    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hp67\" (UniqueName: \"kubernetes.io/projected/d8842256-47fc-4dc5-91f6-65938f410080-kube-api-access-5hp67\") pod \"kas-connection-checker-685895c7c9-wl4tz\" (UID: \"d8842256-47fc-4dc5-91f6-65938f410080\") " pod="kube-system/kas-connection-checker-685895c7c9-wl4tz"
Oct 30 18:36:27.475164 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.475138    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hq7p\" (UniqueName: \"kubernetes.io/projected/e438abba-e5b6-41bb-bc67-1a1d322e6534-kube-api-access-6hq7p\") pod \"kas-connection-checker-685895c7c9-f4bz7\" (UID: \"e438abba-e5b6-41bb-bc67-1a1d322e6534\") " pod="kube-system/kas-connection-checker-685895c7c9-f4bz7"
Oct 30 18:36:27.475989 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.475962    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knktl\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-kube-api-access-knktl\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:27.476172 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.476155    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dth6b\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-kube-api-access-dth6b\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:27.476567 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.476552    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xg8f\" (UniqueName: \"kubernetes.io/projected/5d05b776-9750-4c5b-8d91-4cf27329a1fc-kube-api-access-2xg8f\") pod \"volume-data-source-validator-58c9f5f46d-rpvxs\" (UID: \"5d05b776-9750-4c5b-8d91-4cf27329a1fc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rpvxs"
Oct 30 18:36:27.476823 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.476807    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-bound-sa-token\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:27.509300 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.509283    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-f4bz7"
Oct 30 18:36:27.514567 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.514552    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wl4tz"
Oct 30 18:36:27.517323 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.517211    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-default-certificate\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:27.517323 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.517240    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:27.517323 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.517267    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmg9f\" (UniqueName: \"kubernetes.io/projected/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-kube-api-access-dmg9f\") pod \"cluster-samples-operator-6ddb6c8b94-2xb8v\" (UID: \"b94a90ae-e06a-4e60-a48e-b567ce6e4edd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"
Oct 30 18:36:27.517323 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.517295    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmjgg\" (UniqueName: \"kubernetes.io/projected/05fe2f04-29f7-4999-87ef-b30035b34883-kube-api-access-mmjgg\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:27.517323 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.517322    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12928ad7-4038-473a-a4aa-be1a6a42eabc-tmp-dir\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:27.517570 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.517348    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0648f559-00ac-4de0-8b9f-ce02d1b657a9-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk\" (UID: \"0648f559-00ac-4de0-8b9f-ce02d1b657a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk"
Oct 30 18:36:27.517570 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.517386    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e954d56b-ace6-4539-a467-486c1e889886-config\") pod \"service-ca-operator-544d749c8c-wpnq5\" (UID: \"e954d56b-ace6-4539-a467-486c1e889886\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wpnq5"
Oct 30 18:36:27.517570 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.517417    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-service-ca-bundle\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:27.517570 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.517443    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcz45\" (UniqueName: \"kubernetes.io/projected/12928ad7-4038-473a-a4aa-be1a6a42eabc-kube-api-access-qcz45\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:27.517570 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.517470    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-stats-auth\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:27.517979 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.517941    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9b63e008-3071-4593-bba8-3bcc673b5dc2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6pktt\" (UID: \"9b63e008-3071-4593-bba8-3bcc673b5dc2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6pktt"
Oct 30 18:36:27.518139 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.518122    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e954d56b-ace6-4539-a467-486c1e889886-config\") pod \"service-ca-operator-544d749c8c-wpnq5\" (UID: \"e954d56b-ace6-4539-a467-486c1e889886\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wpnq5"
Oct 30 18:36:27.518231 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.518125    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-service-ca-bundle\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:27.518231 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.518173    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12928ad7-4038-473a-a4aa-be1a6a42eabc-tmp-dir\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:27.520040 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.518400    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21b6f672-2dc5-4523-9dfa-6dfaa30a8bac-cert\") pod \"ingress-canary-g6sxb\" (UID: \"21b6f672-2dc5-4523-9dfa-6dfaa30a8bac\") " pod="openshift-ingress-canary/ingress-canary-g6sxb"
Oct 30 18:36:27.520040 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.518441    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-snapshots\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:27.520040 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.518474    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e954d56b-ace6-4539-a467-486c1e889886-serving-cert\") pod \"service-ca-operator-544d749c8c-wpnq5\" (UID: \"e954d56b-ace6-4539-a467-486c1e889886\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wpnq5"
Oct 30 18:36:27.520040 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.518485    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:27.520040 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.518502    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks7sd\" (UniqueName: \"kubernetes.io/projected/21b6f672-2dc5-4523-9dfa-6dfaa30a8bac-kube-api-access-ks7sd\") pod \"ingress-canary-g6sxb\" (UID: \"21b6f672-2dc5-4523-9dfa-6dfaa30a8bac\") " pod="openshift-ingress-canary/ingress-canary-g6sxb"
Oct 30 18:36:27.520040 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.518528    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-serving-cert\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:27.520040 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.518552    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:27.520040 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.518576    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0648f559-00ac-4de0-8b9f-ce02d1b657a9-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk\" (UID: \"0648f559-00ac-4de0-8b9f-ce02d1b657a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk"
Oct 30 18:36:27.520040 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.518603    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-tmp\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:27.520040 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.518644    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2xb8v\" (UID: \"b94a90ae-e06a-4e60-a48e-b567ce6e4edd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"
Oct 30 18:36:27.520040 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.518662    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0648f559-00ac-4de0-8b9f-ce02d1b657a9-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk\" (UID: \"0648f559-00ac-4de0-8b9f-ce02d1b657a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk"
Oct 30 18:36:27.520040 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.518669    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12928ad7-4038-473a-a4aa-be1a6a42eabc-config-volume\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:27.520040 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.518711    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60e0e397-bb2c-4637-9376-c5af871666cb-trusted-ca\") pod \"console-operator-7cd56b8dbc-jkgcq\" (UID: \"60e0e397-bb2c-4637-9376-c5af871666cb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:27.520040 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.518737    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8ggz\" (UniqueName: \"kubernetes.io/projected/0648f559-00ac-4de0-8b9f-ce02d1b657a9-kube-api-access-k8ggz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk\" (UID: \"0648f559-00ac-4de0-8b9f-ce02d1b657a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk"
Oct 30 18:36:27.520040 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.518771    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60e0e397-bb2c-4637-9376-c5af871666cb-serving-cert\") pod \"console-operator-7cd56b8dbc-jkgcq\" (UID: \"60e0e397-bb2c-4637-9376-c5af871666cb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:27.520040 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.518814    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/60e0e397-bb2c-4637-9376-c5af871666cb-config\") pod \"console-operator-7cd56b8dbc-jkgcq\" (UID: \"60e0e397-bb2c-4637-9376-c5af871666cb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:27.520846 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.518840    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6pktt\" (UID: \"9b63e008-3071-4593-bba8-3bcc673b5dc2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6pktt"
Oct 30 18:36:27.520846 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.518866    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qznzq\" (UniqueName: \"kubernetes.io/projected/e954d56b-ace6-4539-a467-486c1e889886-kube-api-access-qznzq\") pod \"service-ca-operator-544d749c8c-wpnq5\" (UID: \"e954d56b-ace6-4539-a467-486c1e889886\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wpnq5"
Oct 30 18:36:27.520846 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.518900    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-2xb8v\" (UID: \"b94a90ae-e06a-4e60-a48e-b567ce6e4edd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"
Oct 30 18:36:27.520846 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.518927    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx8hm\" (UniqueName: \"kubernetes.io/projected/5ef02004-71fc-4464-9550-bd094d27f21f-kube-api-access-dx8hm\") pod \"network-check-source-6f78d59f99-fswdh\" (UID: \"5ef02004-71fc-4464-9550-bd094d27f21f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fswdh"
Oct 30 18:36:27.520846 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.518940    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9b63e008-3071-4593-bba8-3bcc673b5dc2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6pktt\" (UID: \"9b63e008-3071-4593-bba8-3bcc673b5dc2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6pktt"
Oct 30 18:36:27.520846 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.518951    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:27.520846 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.518996    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v526v\" (UniqueName: \"kubernetes.io/projected/60e0e397-bb2c-4637-9376-c5af871666cb-kube-api-access-v526v\") pod \"console-operator-7cd56b8dbc-jkgcq\" (UID: \"60e0e397-bb2c-4637-9376-c5af871666cb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:27.520846 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.519044    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l74z8\" (UniqueName: \"kubernetes.io/projected/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-kube-api-access-l74z8\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:27.520846 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.519097    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:27.520846 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.519152    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12928ad7-4038-473a-a4aa-be1a6a42eabc-config-volume\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:27.520846 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:27.519476    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:36:27.520846 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:27.519534    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert podName:9b63e008-3071-4593-bba8-3bcc673b5dc2 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:28.019515356 +0000 UTC m=+31.199537469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6pktt" (UID: "9b63e008-3071-4593-bba8-3bcc673b5dc2") : secret "networking-console-plugin-cert" not found
Oct 30 18:36:27.520846 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.519668    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60e0e397-bb2c-4637-9376-c5af871666cb-trusted-ca\") pod \"console-operator-7cd56b8dbc-jkgcq\" (UID: \"60e0e397-bb2c-4637-9376-c5af871666cb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:27.520846 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.519728    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-snapshots\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:27.520846 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.520048    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-default-certificate\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:27.520846 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:27.520174    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:27.521659 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:27.520218    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls podName:12928ad7-4038-473a-a4aa-be1a6a42eabc nodeName:}" failed. No retries permitted until 2026-10-30 18:36:28.02020359 +0000 UTC m=+31.200225704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls") pod "dns-default-sxk64" (UID: "12928ad7-4038-473a-a4aa-be1a6a42eabc") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:27.521659 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.520505    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-2xb8v\" (UID: \"b94a90ae-e06a-4e60-a48e-b567ce6e4edd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"
Oct 30 18:36:27.521659 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.520953    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-tmp\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:27.521659 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.521284    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21b6f672-2dc5-4523-9dfa-6dfaa30a8bac-cert\") pod \"ingress-canary-g6sxb\" (UID: \"21b6f672-2dc5-4523-9dfa-6dfaa30a8bac\") " pod="openshift-ingress-canary/ingress-canary-g6sxb"
Oct 30 18:36:27.521659 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:27.521370    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:36:27.521659 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:27.521375    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:36:27.521659 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:27.521427    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls podName:b94a90ae-e06a-4e60-a48e-b567ce6e4edd nodeName:}" failed. No retries permitted until 2026-10-30 18:36:28.021410751 +0000 UTC m=+31.201432863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2xb8v" (UID: "b94a90ae-e06a-4e60-a48e-b567ce6e4edd") : secret "samples-operator-tls" not found
Oct 30 18:36:27.521659 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:27.521449    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs podName:05fe2f04-29f7-4999-87ef-b30035b34883 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:28.02143875 +0000 UTC m=+31.201460855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs") pod "router-default-b44dd54dc-hqb7p" (UID: "05fe2f04-29f7-4999-87ef-b30035b34883") : secret "router-metrics-certs-default" not found
Oct 30 18:36:27.521659 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:27.521480    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle podName:05fe2f04-29f7-4999-87ef-b30035b34883 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:28.021463647 +0000 UTC m=+31.201485778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle") pod "router-default-b44dd54dc-hqb7p" (UID: "05fe2f04-29f7-4999-87ef-b30035b34883") : configmap references non-existent config key: service-ca.crt
Oct 30 18:36:27.521659 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.521537    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-65rz2"
Oct 30 18:36:27.522188 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.521993    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/60e0e397-bb2c-4637-9376-c5af871666cb-config\") pod \"console-operator-7cd56b8dbc-jkgcq\" (UID: \"60e0e397-bb2c-4637-9376-c5af871666cb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:27.522898 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.522857    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60e0e397-bb2c-4637-9376-c5af871666cb-serving-cert\") pod \"console-operator-7cd56b8dbc-jkgcq\" (UID: \"60e0e397-bb2c-4637-9376-c5af871666cb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:27.524300 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.524155    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0648f559-00ac-4de0-8b9f-ce02d1b657a9-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk\" (UID: \"0648f559-00ac-4de0-8b9f-ce02d1b657a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk"
Oct 30 18:36:27.524300 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.524206    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e954d56b-ace6-4539-a467-486c1e889886-serving-cert\") pod \"service-ca-operator-544d749c8c-wpnq5\" (UID: \"e954d56b-ace6-4539-a467-486c1e889886\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wpnq5"
Oct 30 18:36:27.525042 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.525017    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-serving-cert\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:27.525821 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.525799    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-stats-auth\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:27.579270 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.576999    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rpvxs"
Oct 30 18:36:27.582371 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.581690    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcz45\" (UniqueName: \"kubernetes.io/projected/12928ad7-4038-473a-a4aa-be1a6a42eabc-kube-api-access-qcz45\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:27.583967 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.583308    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmjgg\" (UniqueName: \"kubernetes.io/projected/05fe2f04-29f7-4999-87ef-b30035b34883-kube-api-access-mmjgg\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:27.583967 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.583699    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l74z8\" (UniqueName: \"kubernetes.io/projected/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-kube-api-access-l74z8\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:27.583967 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.583810    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v526v\" (UniqueName: \"kubernetes.io/projected/60e0e397-bb2c-4637-9376-c5af871666cb-kube-api-access-v526v\") pod \"console-operator-7cd56b8dbc-jkgcq\" (UID: \"60e0e397-bb2c-4637-9376-c5af871666cb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:27.587804 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.587541    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks7sd\" (UniqueName: \"kubernetes.io/projected/21b6f672-2dc5-4523-9dfa-6dfaa30a8bac-kube-api-access-ks7sd\") pod \"ingress-canary-g6sxb\" (UID: \"21b6f672-2dc5-4523-9dfa-6dfaa30a8bac\") " pod="openshift-ingress-canary/ingress-canary-g6sxb"
Oct 30 18:36:27.587804 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.587543    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8ggz\" (UniqueName: \"kubernetes.io/projected/0648f559-00ac-4de0-8b9f-ce02d1b657a9-kube-api-access-k8ggz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk\" (UID: \"0648f559-00ac-4de0-8b9f-ce02d1b657a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk"
Oct 30 18:36:27.587804 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.587645    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmg9f\" (UniqueName: \"kubernetes.io/projected/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-kube-api-access-dmg9f\") pod \"cluster-samples-operator-6ddb6c8b94-2xb8v\" (UID: \"b94a90ae-e06a-4e60-a48e-b567ce6e4edd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"
Oct 30 18:36:27.587976 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.587922    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx8hm\" (UniqueName: \"kubernetes.io/projected/5ef02004-71fc-4464-9550-bd094d27f21f-kube-api-access-dx8hm\") pod \"network-check-source-6f78d59f99-fswdh\" (UID: \"5ef02004-71fc-4464-9550-bd094d27f21f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fswdh"
Oct 30 18:36:27.593289 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.593237    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qznzq\" (UniqueName: \"kubernetes.io/projected/e954d56b-ace6-4539-a467-486c1e889886-kube-api-access-qznzq\") pod \"service-ca-operator-544d749c8c-wpnq5\" (UID: \"e954d56b-ace6-4539-a467-486c1e889886\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wpnq5"
Oct 30 18:36:27.606474 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.606083    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:27.617005 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.616844    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wpnq5"
Oct 30 18:36:27.623195 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.623047    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk"
Oct 30 18:36:27.657831 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.656991    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fswdh"
Oct 30 18:36:27.669049 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.668853    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g6sxb"
Oct 30 18:36:27.881091 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.881051    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:27.893799 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:36:27.893757    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda20a00eb_cde2_4df4_a191_0502fcaf288d.slice/crio-bf414281ada08cae7668511c127437eedad35ede1b4ec220a53de48da3f35ebe WatchSource:0}: Error finding container bf414281ada08cae7668511c127437eedad35ede1b4ec220a53de48da3f35ebe: Status 404 returned error can't find the container with id bf414281ada08cae7668511c127437eedad35ede1b4ec220a53de48da3f35ebe
Oct 30 18:36:27.894627 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.894606    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-65rz2"]
Oct 30 18:36:27.896877 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.896852    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wl4tz"]
Oct 30 18:36:27.897204 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:36:27.897177    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8842256_47fc_4dc5_91f6_65938f410080.slice/crio-f46bb25af1cc8f5165ee09c930e7a01203af203a8515498da17f95919c2cc4f4 WatchSource:0}: Error finding container f46bb25af1cc8f5165ee09c930e7a01203af203a8515498da17f95919c2cc4f4: Status 404 returned error can't find the container with id f46bb25af1cc8f5165ee09c930e7a01203af203a8515498da17f95919c2cc4f4
Oct 30 18:36:27.903229 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.903187    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-f4bz7"]
Oct 30 18:36:27.903589 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:36:27.903557    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode438abba_e5b6_41bb_bc67_1a1d322e6534.slice/crio-2e099f4ea8091b1a7d81d6f31319a778089bcb24400d3e279d44461982cda02d WatchSource:0}: Error finding container 2e099f4ea8091b1a7d81d6f31319a778089bcb24400d3e279d44461982cda02d: Status 404 returned error can't find the container with id 2e099f4ea8091b1a7d81d6f31319a778089bcb24400d3e279d44461982cda02d
Oct 30 18:36:27.926876 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.922123    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-plkxx\" (UID: \"80e45018-1682-47cd-9523-dbeaf35a39cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:27.926876 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.922189    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:27.926876 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.922286    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:27.926876 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:27.922437    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:27.926876 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:27.922451    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8598c87c5b-d4m59: secret "image-registry-tls" not found
Oct 30 18:36:27.926876 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:27.922504    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls podName:92981ab7-a41b-482f-98c3-a328e5e8d6f0 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:28.9224878 +0000 UTC m=+32.102509899 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls") pod "image-registry-8598c87c5b-d4m59" (UID: "92981ab7-a41b-482f-98c3-a328e5e8d6f0") : secret "image-registry-tls" not found
Oct 30 18:36:27.926876 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:27.922568    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:27.926876 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:27.922749    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls podName:80e45018-1682-47cd-9523-dbeaf35a39cd nodeName:}" failed. No retries permitted until 2026-10-30 18:36:28.92273342 +0000 UTC m=+32.102755520 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-plkxx" (UID: "80e45018-1682-47cd-9523-dbeaf35a39cd") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:27.926876 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:27.923375    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:27.926876 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:27.923398    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b777d677-gcvwt: secret "image-registry-tls" not found
Oct 30 18:36:27.926876 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:27.923440    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls podName:be409e3c-3420-4250-bfa7-3b01ecd69d9b nodeName:}" failed. No retries permitted until 2026-10-30 18:36:28.923425941 +0000 UTC m=+32.103448042 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls") pod "image-registry-7b777d677-gcvwt" (UID: "be409e3c-3420-4250-bfa7-3b01ecd69d9b") : secret "image-registry-tls" not found
Oct 30 18:36:27.960455 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.960423    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rpvxs"]
Oct 30 18:36:27.960761 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:36:27.960736    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d05b776_9750_4c5b_8d91_4cf27329a1fc.slice/crio-a0551ed10d5edfadfa22b5ba6600154ed5ee9ff3090c925802c7867c7e97bfec WatchSource:0}: Error finding container a0551ed10d5edfadfa22b5ba6600154ed5ee9ff3090c925802c7867c7e97bfec: Status 404 returned error can't find the container with id a0551ed10d5edfadfa22b5ba6600154ed5ee9ff3090c925802c7867c7e97bfec
Oct 30 18:36:27.987079 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:27.985734    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"]
Oct 30 18:36:28.006312 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.006288    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk"]
Oct 30 18:36:28.006515 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:36:28.006491    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0648f559_00ac_4de0_8b9f_ce02d1b657a9.slice/crio-8ecb952775b583a3883cc36ec2b142c98891e7d39ff126a4edcd4a1eef4131f3 WatchSource:0}: Error finding container 8ecb952775b583a3883cc36ec2b142c98891e7d39ff126a4edcd4a1eef4131f3: Status 404 returned error can't find the container with id 8ecb952775b583a3883cc36ec2b142c98891e7d39ff126a4edcd4a1eef4131f3
Oct 30 18:36:28.008683 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.008619    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-wpnq5"]
Oct 30 18:36:28.009124 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:36:28.009099    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode954d56b_ace6_4539_a467_486c1e889886.slice/crio-14de059988139725a952bc59f04bd0c84c5561ad20b6de9d807f2577d2cb131a WatchSource:0}: Error finding container 14de059988139725a952bc59f04bd0c84c5561ad20b6de9d807f2577d2cb131a: Status 404 returned error can't find the container with id 14de059988139725a952bc59f04bd0c84c5561ad20b6de9d807f2577d2cb131a
Oct 30 18:36:28.022793 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.022771    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:28.022860 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.022812    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2xb8v\" (UID: \"b94a90ae-e06a-4e60-a48e-b567ce6e4edd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"
Oct 30 18:36:28.022860 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.022849    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6pktt\" (UID: \"9b63e008-3071-4593-bba8-3bcc673b5dc2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6pktt"
Oct 30 18:36:28.022932 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.022872    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:28.022932 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.022892    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:28.022932 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:28.022899    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:28.023013 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:28.022943    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls podName:12928ad7-4038-473a-a4aa-be1a6a42eabc nodeName:}" failed. No retries permitted until 2026-10-30 18:36:29.022929233 +0000 UTC m=+32.202951332 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls") pod "dns-default-sxk64" (UID: "12928ad7-4038-473a-a4aa-be1a6a42eabc") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:28.023013 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:28.022980    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:36:28.023013 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:28.023012    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs podName:05fe2f04-29f7-4999-87ef-b30035b34883 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:29.023002607 +0000 UTC m=+32.203024721 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs") pod "router-default-b44dd54dc-hqb7p" (UID: "05fe2f04-29f7-4999-87ef-b30035b34883") : secret "router-metrics-certs-default" not found
Oct 30 18:36:28.023159 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:28.023025    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:36:28.023159 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:28.023048    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert podName:9b63e008-3071-4593-bba8-3bcc673b5dc2 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:29.023041675 +0000 UTC m=+32.203063774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6pktt" (UID: "9b63e008-3071-4593-bba8-3bcc673b5dc2") : secret "networking-console-plugin-cert" not found
Oct 30 18:36:28.023159 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:28.023059    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle podName:05fe2f04-29f7-4999-87ef-b30035b34883 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:29.023052888 +0000 UTC m=+32.203075018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle") pod "router-default-b44dd54dc-hqb7p" (UID: "05fe2f04-29f7-4999-87ef-b30035b34883") : configmap references non-existent config key: service-ca.crt
Oct 30 18:36:28.023159 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:28.023102    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:36:28.023159 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:28.023122    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls podName:b94a90ae-e06a-4e60-a48e-b567ce6e4edd nodeName:}" failed. No retries permitted until 2026-10-30 18:36:29.023115884 +0000 UTC m=+32.203138001 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2xb8v" (UID: "b94a90ae-e06a-4e60-a48e-b567ce6e4edd") : secret "samples-operator-tls" not found
Oct 30 18:36:28.055796 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.054937    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g6sxb"]
Oct 30 18:36:28.057142 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.056927    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fswdh"]
Oct 30 18:36:28.058204 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:36:28.058178    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b6f672_2dc5_4523_9dfa_6dfaa30a8bac.slice/crio-aaffbe47de8f1818df45a736effd1ad88bf5a61518d31a7a73b2695a9f07e400 WatchSource:0}: Error finding container aaffbe47de8f1818df45a736effd1ad88bf5a61518d31a7a73b2695a9f07e400: Status 404 returned error can't find the container with id aaffbe47de8f1818df45a736effd1ad88bf5a61518d31a7a73b2695a9f07e400
Oct 30 18:36:28.059433 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:36:28.059415    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef02004_71fc_4464_9550_bd094d27f21f.slice/crio-6b7dd88b77a31d2d61979b2e9b046603cc78f3aae97bc95a0bac4ddbf3309384 WatchSource:0}: Error finding container 6b7dd88b77a31d2d61979b2e9b046603cc78f3aae97bc95a0bac4ddbf3309384: Status 404 returned error can't find the container with id 6b7dd88b77a31d2d61979b2e9b046603cc78f3aae97bc95a0bac4ddbf3309384
Oct 30 18:36:28.267029 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:36:28.266995    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb5e9b3_82ee_4bbe_bc5f_8c763d6120e9.slice/crio-f8e7bbb9d66216640a2e4edc5ffdb3166448ec13bc4800d1523680d155f3634b WatchSource:0}: Error finding container f8e7bbb9d66216640a2e4edc5ffdb3166448ec13bc4800d1523680d155f3634b: Status 404 returned error can't find the container with id f8e7bbb9d66216640a2e4edc5ffdb3166448ec13bc4800d1523680d155f3634b
Oct 30 18:36:28.267753 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.267732    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-776vb"]
Oct 30 18:36:28.307356 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.307331    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:36:28.361445 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.361416    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5slhp\""
Oct 30 18:36:28.486226 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.486194    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq" event={"ID":"60e0e397-bb2c-4637-9376-c5af871666cb","Type":"ContainerStarted","Data":"3aaf547da0a47eca32ce390a3406034d7022e20fea73e56992f72657a781c2dd"}
Oct 30 18:36:28.487596 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.487570    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-65rz2" event={"ID":"a20a00eb-cde2-4df4-a191-0502fcaf288d","Type":"ContainerStarted","Data":"07a5a74b74a3d3f86ffc98b5787f979e222c15e6629d47da37e06e73277d60aa"}
Oct 30 18:36:28.487700 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.487608    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-65rz2" event={"ID":"a20a00eb-cde2-4df4-a191-0502fcaf288d","Type":"ContainerStarted","Data":"bf414281ada08cae7668511c127437eedad35ede1b4ec220a53de48da3f35ebe"}
Oct 30 18:36:28.489113 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.489087    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wl4tz" event={"ID":"d8842256-47fc-4dc5-91f6-65938f410080","Type":"ContainerStarted","Data":"c7d4cf2e919c8527645cc5d614eca3062ef59ef16a7fde965fa731b724f18660"}
Oct 30 18:36:28.489201 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.489126    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wl4tz" event={"ID":"d8842256-47fc-4dc5-91f6-65938f410080","Type":"ContainerStarted","Data":"f46bb25af1cc8f5165ee09c930e7a01203af203a8515498da17f95919c2cc4f4"}
Oct 30 18:36:28.490422 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.490397    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-f4bz7" event={"ID":"e438abba-e5b6-41bb-bc67-1a1d322e6534","Type":"ContainerStarted","Data":"31329fb1aa8a5d31b38fed98a9bb15258211c0b280c3b887157164e6aabe7b50"}
Oct 30 18:36:28.490529 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.490436    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-f4bz7" event={"ID":"e438abba-e5b6-41bb-bc67-1a1d322e6534","Type":"ContainerStarted","Data":"2e099f4ea8091b1a7d81d6f31319a778089bcb24400d3e279d44461982cda02d"}
Oct 30 18:36:28.491475 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.491457    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-776vb" event={"ID":"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9","Type":"ContainerStarted","Data":"f8e7bbb9d66216640a2e4edc5ffdb3166448ec13bc4800d1523680d155f3634b"}
Oct 30 18:36:28.492440 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.492424    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk" event={"ID":"0648f559-00ac-4de0-8b9f-ce02d1b657a9","Type":"ContainerStarted","Data":"8ecb952775b583a3883cc36ec2b142c98891e7d39ff126a4edcd4a1eef4131f3"}
Oct 30 18:36:28.493409 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.493379    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rpvxs" event={"ID":"5d05b776-9750-4c5b-8d91-4cf27329a1fc","Type":"ContainerStarted","Data":"a0551ed10d5edfadfa22b5ba6600154ed5ee9ff3090c925802c7867c7e97bfec"}
Oct 30 18:36:28.494424 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.494405    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fswdh" event={"ID":"5ef02004-71fc-4464-9550-bd094d27f21f","Type":"ContainerStarted","Data":"6b7dd88b77a31d2d61979b2e9b046603cc78f3aae97bc95a0bac4ddbf3309384"}
Oct 30 18:36:28.495463 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.495444    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g6sxb" event={"ID":"21b6f672-2dc5-4523-9dfa-6dfaa30a8bac","Type":"ContainerStarted","Data":"aaffbe47de8f1818df45a736effd1ad88bf5a61518d31a7a73b2695a9f07e400"}
Oct 30 18:36:28.496376 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.496360    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wpnq5" event={"ID":"e954d56b-ace6-4539-a467-486c1e889886","Type":"ContainerStarted","Data":"14de059988139725a952bc59f04bd0c84c5561ad20b6de9d807f2577d2cb131a"}
Oct 30 18:36:28.931798 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.931764    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:28.931977 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.931886    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-plkxx\" (UID: \"80e45018-1682-47cd-9523-dbeaf35a39cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:28.931977 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:28.931911    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:28.931977 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:28.931932    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8598c87c5b-d4m59: secret "image-registry-tls" not found
Oct 30 18:36:28.931977 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:28.931953    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:28.932198 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:28.931988    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls podName:92981ab7-a41b-482f-98c3-a328e5e8d6f0 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:30.931970439 +0000 UTC m=+34.111992553 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls") pod "image-registry-8598c87c5b-d4m59" (UID: "92981ab7-a41b-482f-98c3-a328e5e8d6f0") : secret "image-registry-tls" not found
Oct 30 18:36:28.932198 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:28.932047    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:28.932198 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:28.932060    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b777d677-gcvwt: secret "image-registry-tls" not found
Oct 30 18:36:28.932198 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:28.932128    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls podName:be409e3c-3420-4250-bfa7-3b01ecd69d9b nodeName:}" failed. No retries permitted until 2026-10-30 18:36:30.932102759 +0000 UTC m=+34.112124862 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls") pod "image-registry-7b777d677-gcvwt" (UID: "be409e3c-3420-4250-bfa7-3b01ecd69d9b") : secret "image-registry-tls" not found
Oct 30 18:36:28.932198 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:28.932140    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:28.932198 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:28.932170    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls podName:80e45018-1682-47cd-9523-dbeaf35a39cd nodeName:}" failed. No retries permitted until 2026-10-30 18:36:30.932162563 +0000 UTC m=+34.112184687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-plkxx" (UID: "80e45018-1682-47cd-9523-dbeaf35a39cd") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:29.033699 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:29.033647    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6pktt\" (UID: \"9b63e008-3071-4593-bba8-3bcc673b5dc2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6pktt"
Oct 30 18:36:29.033863 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:29.033730    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:29.033863 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:29.033759    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:29.034073 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:29.034044    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:29.034141 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:29.034116    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2xb8v\" (UID: \"b94a90ae-e06a-4e60-a48e-b567ce6e4edd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"
Oct 30 18:36:29.040266 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:29.040244    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle podName:05fe2f04-29f7-4999-87ef-b30035b34883 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:31.040214312 +0000 UTC m=+34.220236425 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle") pod "router-default-b44dd54dc-hqb7p" (UID: "05fe2f04-29f7-4999-87ef-b30035b34883") : configmap references non-existent config key: service-ca.crt
Oct 30 18:36:29.040397 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:29.040374    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:36:29.040454 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:29.040434    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert podName:9b63e008-3071-4593-bba8-3bcc673b5dc2 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:31.040412755 +0000 UTC m=+34.220434869 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6pktt" (UID: "9b63e008-3071-4593-bba8-3bcc673b5dc2") : secret "networking-console-plugin-cert" not found
Oct 30 18:36:29.040741 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:29.040514    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:36:29.040741 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:29.040558    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs podName:05fe2f04-29f7-4999-87ef-b30035b34883 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:31.040546953 +0000 UTC m=+34.220569067 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs") pod "router-default-b44dd54dc-hqb7p" (UID: "05fe2f04-29f7-4999-87ef-b30035b34883") : secret "router-metrics-certs-default" not found
Oct 30 18:36:29.040741 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:29.040640    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:29.040741 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:29.040683    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls podName:12928ad7-4038-473a-a4aa-be1a6a42eabc nodeName:}" failed. No retries permitted until 2026-10-30 18:36:31.040671182 +0000 UTC m=+34.220693296 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls") pod "dns-default-sxk64" (UID: "12928ad7-4038-473a-a4aa-be1a6a42eabc") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:29.044105 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:29.041129    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:36:29.044105 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:29.041196    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls podName:b94a90ae-e06a-4e60-a48e-b567ce6e4edd nodeName:}" failed. No retries permitted until 2026-10-30 18:36:31.041180086 +0000 UTC m=+34.221202201 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2xb8v" (UID: "b94a90ae-e06a-4e60-a48e-b567ce6e4edd") : secret "samples-operator-tls" not found
Oct 30 18:36:29.419923 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:29.419869    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-65rz2" podStartSLOduration=433.419853291 podStartE2EDuration="7m13.419853291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:29.418695264 +0000 UTC m=+32.598717386" watchObservedRunningTime="2026-10-30 18:36:29.419853291 +0000 UTC m=+32.599875416"
Oct 30 18:36:29.649976 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:29.649919    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-f4bz7" podStartSLOduration=433.64989912 podStartE2EDuration="7m13.64989912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:29.534613801 +0000 UTC m=+32.714635923" watchObservedRunningTime="2026-10-30 18:36:29.64989912 +0000 UTC m=+32.829921242"
Oct 30 18:36:29.786581 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:29.786454    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:30.079943 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:30.079652    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:30.963819 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:30.962774    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-plkxx\" (UID: \"80e45018-1682-47cd-9523-dbeaf35a39cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:30.963819 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:30.962847    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:30.963819 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:30.962944    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:30.963819 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:30.963107    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:30.963819 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:30.963123    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8598c87c5b-d4m59: secret "image-registry-tls" not found
Oct 30 18:36:30.963819 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:30.963182    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls podName:92981ab7-a41b-482f-98c3-a328e5e8d6f0 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:34.963163697 +0000 UTC m=+38.143185810 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls") pod "image-registry-8598c87c5b-d4m59" (UID: "92981ab7-a41b-482f-98c3-a328e5e8d6f0") : secret "image-registry-tls" not found
Oct 30 18:36:30.963819 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:30.963601    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:30.963819 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:30.963657    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls podName:80e45018-1682-47cd-9523-dbeaf35a39cd nodeName:}" failed. No retries permitted until 2026-10-30 18:36:34.963642367 +0000 UTC m=+38.143664471 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-plkxx" (UID: "80e45018-1682-47cd-9523-dbeaf35a39cd") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:30.963819 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:30.963718    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:30.963819 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:30.963728    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b777d677-gcvwt: secret "image-registry-tls" not found
Oct 30 18:36:30.963819 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:30.963760    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls podName:be409e3c-3420-4250-bfa7-3b01ecd69d9b nodeName:}" failed. No retries permitted until 2026-10-30 18:36:34.963747736 +0000 UTC m=+38.143769838 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls") pod "image-registry-7b777d677-gcvwt" (UID: "be409e3c-3420-4250-bfa7-3b01ecd69d9b") : secret "image-registry-tls" not found
Oct 30 18:36:31.064311 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:31.064277    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2xb8v\" (UID: \"b94a90ae-e06a-4e60-a48e-b567ce6e4edd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"
Oct 30 18:36:31.064423 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:31.064365    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6pktt\" (UID: \"9b63e008-3071-4593-bba8-3bcc673b5dc2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6pktt"
Oct 30 18:36:31.064486 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:31.064407    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:31.064486 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:31.064456    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:31.064595 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:31.064568    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:31.064722 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:31.064706    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:31.064785 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:31.064773    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls podName:12928ad7-4038-473a-a4aa-be1a6a42eabc nodeName:}" failed. No retries permitted until 2026-10-30 18:36:35.064755386 +0000 UTC m=+38.244777489 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls") pod "dns-default-sxk64" (UID: "12928ad7-4038-473a-a4aa-be1a6a42eabc") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:31.064846 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:31.064834    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:36:31.064887 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:31.064872    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls podName:b94a90ae-e06a-4e60-a48e-b567ce6e4edd nodeName:}" failed. No retries permitted until 2026-10-30 18:36:35.064861719 +0000 UTC m=+38.244883838 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2xb8v" (UID: "b94a90ae-e06a-4e60-a48e-b567ce6e4edd") : secret "samples-operator-tls" not found
Oct 30 18:36:31.064934 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:31.064929    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:36:31.064983 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:31.064958    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert podName:9b63e008-3071-4593-bba8-3bcc673b5dc2 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:35.064949387 +0000 UTC m=+38.244971490 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6pktt" (UID: "9b63e008-3071-4593-bba8-3bcc673b5dc2") : secret "networking-console-plugin-cert" not found
Oct 30 18:36:31.065039 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:31.065027    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle podName:05fe2f04-29f7-4999-87ef-b30035b34883 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:35.065017497 +0000 UTC m=+38.245039609 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle") pod "router-default-b44dd54dc-hqb7p" (UID: "05fe2f04-29f7-4999-87ef-b30035b34883") : configmap references non-existent config key: service-ca.crt
Oct 30 18:36:31.065143 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:31.065123    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:36:31.065191 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:31.065159    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs podName:05fe2f04-29f7-4999-87ef-b30035b34883 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:35.065149044 +0000 UTC m=+38.245171157 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs") pod "router-default-b44dd54dc-hqb7p" (UID: "05fe2f04-29f7-4999-87ef-b30035b34883") : secret "router-metrics-certs-default" not found
Oct 30 18:36:31.166380 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:31.165267    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92f38b64-26fb-42d6-826a-94cd37e8b668-original-pull-secret\") pod \"global-pull-secret-syncer-vs8lx\" (UID: \"92f38b64-26fb-42d6-826a-94cd37e8b668\") " pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:31.166380 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:31.165335    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs\") pod \"network-metrics-daemon-sqkfq\" (UID: \"6df0f1e7-07b7-4231-bc70-ba81aef29856\") " pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:31.166380 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:31.165368    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb856\" (UniqueName: \"kubernetes.io/projected/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278-kube-api-access-vb856\") pod \"network-check-target-xcmps\" (UID: \"0b0aa783-d4ab-490c-a5f7-d7da5ad8d278\") " pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:36:31.166380 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:31.165806    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:36:31.166380 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:31.165877    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs podName:6df0f1e7-07b7-4231-bc70-ba81aef29856 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:03.165858512 +0000 UTC m=+66.345880613 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs") pod "network-metrics-daemon-sqkfq" (UID: "6df0f1e7-07b7-4231-bc70-ba81aef29856") : secret "metrics-daemon-secret" not found
Oct 30 18:36:31.174455 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:31.174426    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92f38b64-26fb-42d6-826a-94cd37e8b668-original-pull-secret\") pod \"global-pull-secret-syncer-vs8lx\" (UID: \"92f38b64-26fb-42d6-826a-94cd37e8b668\") " pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:31.177179 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:31.177132    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb856\" (UniqueName: \"kubernetes.io/projected/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278-kube-api-access-vb856\") pod \"network-check-target-xcmps\" (UID: \"0b0aa783-d4ab-490c-a5f7-d7da5ad8d278\") " pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:36:31.253284 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:31.253215    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:31.319891 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:31.319857    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:36:31.516419 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:31.516345    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rpvxs" event={"ID":"5d05b776-9750-4c5b-8d91-4cf27329a1fc","Type":"ContainerStarted","Data":"d59275ceaaaecd7a42314263d705a72e749bc131965e29895a81721005dd79c3"}
Oct 30 18:36:31.633158 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:31.631370    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-wl4tz" podStartSLOduration=435.631350296 podStartE2EDuration="7m15.631350296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:29.650235251 +0000 UTC m=+32.830257381" watchObservedRunningTime="2026-10-30 18:36:31.631350296 +0000 UTC m=+34.811372417"
Oct 30 18:36:31.633158 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:31.632707    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rpvxs" podStartSLOduration=406.556423486 podStartE2EDuration="6m49.632692662s" totalImagesPullingTime="3.076269176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:31.631244506 +0000 UTC m=+34.811266650" watchObservedRunningTime="2026-10-30 18:36:31.632692662 +0000 UTC m=+34.812714784"
Oct 30 18:36:31.665987 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:31.665225    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vs8lx"]
Oct 30 18:36:31.757201 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:31.757162    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xcmps"]
Oct 30 18:36:31.760137 ip-10-0-131-224 kubenswrapper[2572]: W1030 18:36:31.760105    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0aa783_d4ab_490c_a5f7_d7da5ad8d278.slice/crio-b982014ee3a54d7d3cb4c6619b1f3d0b15954c25fd5bac0f95a58c78a3541556 WatchSource:0}: Error finding container b982014ee3a54d7d3cb4c6619b1f3d0b15954c25fd5bac0f95a58c78a3541556: Status 404 returned error can't find the container with id b982014ee3a54d7d3cb4c6619b1f3d0b15954c25fd5bac0f95a58c78a3541556
Oct 30 18:36:31.779495 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:31.779424    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:32.027557 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:32.027513    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:32.522560 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:32.522450    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk" event={"ID":"0648f559-00ac-4de0-8b9f-ce02d1b657a9","Type":"ContainerStarted","Data":"48fc696af2d22b8f8e4aba80ab36337ca1f67a84bd21a7b6810c28457b1f3ed6"}
Oct 30 18:36:32.524517 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:32.524449    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vs8lx" event={"ID":"92f38b64-26fb-42d6-826a-94cd37e8b668","Type":"ContainerStarted","Data":"5edaeb3ad7c0c9d0fe1f145eeace4b609b052a02d7b21055e42ca516feff8f11"}
Oct 30 18:36:32.526257 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:32.526171    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:32.526701 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:32.526634    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xcmps" event={"ID":"0b0aa783-d4ab-490c-a5f7-d7da5ad8d278","Type":"ContainerStarted","Data":"b982014ee3a54d7d3cb4c6619b1f3d0b15954c25fd5bac0f95a58c78a3541556"}
Oct 30 18:36:32.529741 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:32.528470    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/0.log"
Oct 30 18:36:32.529741 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:32.528507    2572 generic.go:356] "Generic (PLEG): container finished" podID="60e0e397-bb2c-4637-9376-c5af871666cb" containerID="4a79150de88aecfbbf39f2bec6c9c46aedeee3db33d59dce57a2de6e94b384d2" exitCode=255
Oct 30 18:36:32.529741 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:32.529242    2572 scope.go:122] "RemoveContainer" containerID="4a79150de88aecfbbf39f2bec6c9c46aedeee3db33d59dce57a2de6e94b384d2"
Oct 30 18:36:32.529741 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:32.529504    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq" event={"ID":"60e0e397-bb2c-4637-9376-c5af871666cb","Type":"ContainerDied","Data":"4a79150de88aecfbbf39f2bec6c9c46aedeee3db33d59dce57a2de6e94b384d2"}
Oct 30 18:36:33.535634 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:33.535604    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/1.log"
Oct 30 18:36:33.536987 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:33.536962    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/0.log"
Oct 30 18:36:33.537111 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:33.537000    2572 generic.go:356] "Generic (PLEG): container finished" podID="60e0e397-bb2c-4637-9376-c5af871666cb" containerID="e7bca663a98eaf8619e4d44e2ab3f7281662e2728ad5d0f643a4a85b0146af10" exitCode=255
Oct 30 18:36:33.538207 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:33.538182    2572 scope.go:122] "RemoveContainer" containerID="e7bca663a98eaf8619e4d44e2ab3f7281662e2728ad5d0f643a4a85b0146af10"
Oct 30 18:36:33.538443 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:33.538408    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jkgcq_openshift-console-operator(60e0e397-bb2c-4637-9376-c5af871666cb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq" podUID="60e0e397-bb2c-4637-9376-c5af871666cb"
Oct 30 18:36:33.538532 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:33.538460    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq" event={"ID":"60e0e397-bb2c-4637-9376-c5af871666cb","Type":"ContainerDied","Data":"e7bca663a98eaf8619e4d44e2ab3f7281662e2728ad5d0f643a4a85b0146af10"}
Oct 30 18:36:33.538532 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:33.538496    2572 scope.go:122] "RemoveContainer" containerID="4a79150de88aecfbbf39f2bec6c9c46aedeee3db33d59dce57a2de6e94b384d2"
Oct 30 18:36:33.655334 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:33.655275    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk" podStartSLOduration=406.241176856 podStartE2EDuration="6m50.655262276s" totalImagesPullingTime="4.41408542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:33.654391239 +0000 UTC m=+36.834413360" watchObservedRunningTime="2026-10-30 18:36:33.655262276 +0000 UTC m=+36.835284396"
Oct 30 18:36:34.141204 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:34.141170    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:34.280260 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:34.279643    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-dfv27"]
Oct 30 18:36:34.294861 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:34.294683    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dfv27"
Oct 30 18:36:34.336523 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:34.336489    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-dfv27"]
Oct 30 18:36:34.374788 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:34.374754    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:34.405209 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:34.405173    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr2vr\" (UniqueName: \"kubernetes.io/projected/9d148c95-78ae-43af-8942-460853970292-kube-api-access-cr2vr\") pod \"migrator-66df5d496b-dfv27\" (UID: \"9d148c95-78ae-43af-8942-460853970292\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dfv27"
Oct 30 18:36:34.423804 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:34.423768    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 18:36:34.423957 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:34.423850    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:34.424928 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:34.424909    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mbxkm\""
Oct 30 18:36:34.506313 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:34.506278    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr2vr\" (UniqueName: \"kubernetes.io/projected/9d148c95-78ae-43af-8942-460853970292-kube-api-access-cr2vr\") pod \"migrator-66df5d496b-dfv27\" (UID: \"9d148c95-78ae-43af-8942-460853970292\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dfv27"
Oct 30 18:36:34.543781 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:34.542776    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/1.log"
Oct 30 18:36:34.543781 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:34.543206    2572 scope.go:122] "RemoveContainer" containerID="e7bca663a98eaf8619e4d44e2ab3f7281662e2728ad5d0f643a4a85b0146af10"
Oct 30 18:36:34.543781 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:34.543419    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jkgcq_openshift-console-operator(60e0e397-bb2c-4637-9376-c5af871666cb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq" podUID="60e0e397-bb2c-4637-9376-c5af871666cb"
Oct 30 18:36:34.545574 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:34.545131    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-776vb" event={"ID":"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9","Type":"ContainerStarted","Data":"9bbb3a662c60e111c0291040c70cbbeee7efa3b3a15e3f0219c5db1145a2c0b6"}
Oct 30 18:36:34.547815 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:34.547416    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g6sxb" event={"ID":"21b6f672-2dc5-4523-9dfa-6dfaa30a8bac","Type":"ContainerStarted","Data":"505ae496f388c8673fc792313ed358b8baed78b6f3733708af20c984c4d130bb"}
Oct 30 18:36:34.549252 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:34.549228    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wpnq5" event={"ID":"e954d56b-ace6-4539-a467-486c1e889886","Type":"ContainerStarted","Data":"9db3b68d4c5b11edd90b0d14940978f3ee6d2e0105105b15cc9a9c7520f86a1f"}
Oct 30 18:36:34.573872 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:34.573850    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr2vr\" (UniqueName: \"kubernetes.io/projected/9d148c95-78ae-43af-8942-460853970292-kube-api-access-cr2vr\") pod \"migrator-66df5d496b-dfv27\" (UID: \"9d148c95-78ae-43af-8942-460853970292\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dfv27"
Oct 30 18:36:34.617029 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:34.616997    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:34.640093 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:34.640039    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dfv27"
Oct 30 18:36:34.780721 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:34.780669    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-g6sxb" podStartSLOduration=1.897137496 podStartE2EDuration="7.780656656s" totalImagesPullingTime="5.88351916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:34.780233597 +0000 UTC m=+37.960255718" watchObservedRunningTime="2026-10-30 18:36:34.780656656 +0000 UTC m=+37.960678774"
Oct 30 18:36:34.800689 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:34.800622    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:34.900885 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:34.900831    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wpnq5" podStartSLOduration=405.996378642 podStartE2EDuration="6m51.900816461s" totalImagesPullingTime="5.904437819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:34.900128335 +0000 UTC m=+38.080150457" watchObservedRunningTime="2026-10-30 18:36:34.900816461 +0000 UTC m=+38.080838572"
Oct 30 18:36:35.013025 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.012136    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:35.013025 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.012252    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-plkxx\" (UID: \"80e45018-1682-47cd-9523-dbeaf35a39cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:35.013025 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.012317    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:35.013025 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:35.012474    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:35.013025 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:35.012489    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b777d677-gcvwt: secret "image-registry-tls" not found
Oct 30 18:36:35.013025 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:35.012546    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls podName:be409e3c-3420-4250-bfa7-3b01ecd69d9b nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.012528418 +0000 UTC m=+46.192550537 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls") pod "image-registry-7b777d677-gcvwt" (UID: "be409e3c-3420-4250-bfa7-3b01ecd69d9b") : secret "image-registry-tls" not found
Oct 30 18:36:35.013025 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:35.012892    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:35.013025 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:35.012954    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls podName:80e45018-1682-47cd-9523-dbeaf35a39cd nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.012937936 +0000 UTC m=+46.192960050 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-plkxx" (UID: "80e45018-1682-47cd-9523-dbeaf35a39cd") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:35.013649 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:35.013629    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:35.013649 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:35.013649    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8598c87c5b-d4m59: secret "image-registry-tls" not found
Oct 30 18:36:35.013815 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:35.013697    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls podName:92981ab7-a41b-482f-98c3-a328e5e8d6f0 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.013682613 +0000 UTC m=+46.193704729 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls") pod "image-registry-8598c87c5b-d4m59" (UID: "92981ab7-a41b-482f-98c3-a328e5e8d6f0") : secret "image-registry-tls" not found
Oct 30 18:36:35.017182 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.017144    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-776vb" podStartSLOduration=407.367399703 podStartE2EDuration="6m53.01713127s" totalImagesPullingTime="5.649731567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:35.015895128 +0000 UTC m=+38.195917251" watchObservedRunningTime="2026-10-30 18:36:35.01713127 +0000 UTC m=+38.197153391"
Oct 30 18:36:35.052057 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.051996    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-dfv27"]
Oct 30 18:36:35.113602 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.113573    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:35.114392 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.113621    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2xb8v\" (UID: \"b94a90ae-e06a-4e60-a48e-b567ce6e4edd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"
Oct 30 18:36:35.114392 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.113678    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6pktt\" (UID: \"9b63e008-3071-4593-bba8-3bcc673b5dc2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6pktt"
Oct 30 18:36:35.114392 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.113712    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:35.114392 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.113744    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:35.114392 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:35.113813    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:36:35.114392 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:35.113883    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:36:35.114392 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:35.113949    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:36:35.114392 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:35.113886    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls podName:b94a90ae-e06a-4e60-a48e-b567ce6e4edd nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.113866871 +0000 UTC m=+46.293888973 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2xb8v" (UID: "b94a90ae-e06a-4e60-a48e-b567ce6e4edd") : secret "samples-operator-tls" not found
Oct 30 18:36:35.114392 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:35.113993    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs podName:05fe2f04-29f7-4999-87ef-b30035b34883 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.113978569 +0000 UTC m=+46.294000667 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs") pod "router-default-b44dd54dc-hqb7p" (UID: "05fe2f04-29f7-4999-87ef-b30035b34883") : secret "router-metrics-certs-default" not found
Oct 30 18:36:35.114392 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:35.114006    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:35.114392 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:35.114016    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle podName:05fe2f04-29f7-4999-87ef-b30035b34883 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.114006645 +0000 UTC m=+46.294028744 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle") pod "router-default-b44dd54dc-hqb7p" (UID: "05fe2f04-29f7-4999-87ef-b30035b34883") : configmap references non-existent config key: service-ca.crt
Oct 30 18:36:35.114392 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:35.114037    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert podName:9b63e008-3071-4593-bba8-3bcc673b5dc2 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.114026735 +0000 UTC m=+46.294048849 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6pktt" (UID: "9b63e008-3071-4593-bba8-3bcc673b5dc2") : secret "networking-console-plugin-cert" not found
Oct 30 18:36:35.114392 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:35.114052    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls podName:12928ad7-4038-473a-a4aa-be1a6a42eabc nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.114043745 +0000 UTC m=+46.294065845 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls") pod "dns-default-sxk64" (UID: "12928ad7-4038-473a-a4aa-be1a6a42eabc") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:35.466840 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.466814    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xcwnk"]
Oct 30 18:36:35.493622 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.493594    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:35.517362 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.517336    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:35.517497 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.517368    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:35.517497 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.517467    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a11d470f-d198-42ec-bee1-a900c4814473-crio-socket\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:35.517660 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.517556    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9z9l\" (UniqueName: \"kubernetes.io/projected/a11d470f-d198-42ec-bee1-a900c4814473-kube-api-access-v9z9l\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:35.517660 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.517607    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a11d470f-d198-42ec-bee1-a900c4814473-data-volume\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:35.530085 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.530042    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xcwnk"]
Oct 30 18:36:35.534307 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.534192    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:35.547019 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.546990    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 18:36:35.547524 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.547140    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 18:36:35.547524 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.547369    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-swngw\""
Oct 30 18:36:35.557380 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.556966    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fswdh" event={"ID":"5ef02004-71fc-4464-9550-bd094d27f21f","Type":"ContainerStarted","Data":"1ae9250e1325873a75aa12a01699e8bd300d4185b6c5849d222208c5823bd663"}
Oct 30 18:36:35.560737 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.560708    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xcmps" event={"ID":"0b0aa783-d4ab-490c-a5f7-d7da5ad8d278","Type":"ContainerStarted","Data":"731ba1e0e728229433927aafaa474aeeecc9e3725aa430a5e67f333f79c15df9"}
Oct 30 18:36:35.560840 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.560782    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:36:35.562471 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.562446    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dfv27" event={"ID":"9d148c95-78ae-43af-8942-460853970292","Type":"ContainerStarted","Data":"022157ce1cf92e0afd69fe9ec69a54f1aa1da7587493d5ce91ef80846f269ccf"}
Oct 30 18:36:35.619115 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.619085    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:35.619280 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.619129    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:35.619280 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.619223    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a11d470f-d198-42ec-bee1-a900c4814473-crio-socket\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:35.619364 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:35.619289    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:35.619364 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.619322    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9z9l\" (UniqueName: \"kubernetes.io/projected/a11d470f-d198-42ec-bee1-a900c4814473-kube-api-access-v9z9l\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:35.619364 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:35.619334    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:35.619454 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:35.619343    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls podName:a11d470f-d198-42ec-bee1-a900c4814473 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:36.119324895 +0000 UTC m=+39.299347018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xcwnk" (UID: "a11d470f-d198-42ec-bee1-a900c4814473") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:35.619454 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:35.619406    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm podName:a11d470f-d198-42ec-bee1-a900c4814473 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:36.119390201 +0000 UTC m=+39.299412307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xcwnk" (UID: "a11d470f-d198-42ec-bee1-a900c4814473") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:35.619523 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.619488    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a11d470f-d198-42ec-bee1-a900c4814473-data-volume\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:35.619598 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.619525    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a11d470f-d198-42ec-bee1-a900c4814473-crio-socket\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:35.619857 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.619830    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a11d470f-d198-42ec-bee1-a900c4814473-data-volume\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:35.681534 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.681501    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9z9l\" (UniqueName: \"kubernetes.io/projected/a11d470f-d198-42ec-bee1-a900c4814473-kube-api-access-v9z9l\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:35.726090 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.726017    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xcmps" podStartSLOduration=34.118204264 podStartE2EDuration="37.725999088s" totalImagesPullingTime="3.607794824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:35.725232489 +0000 UTC m=+38.905254623" watchObservedRunningTime="2026-10-30 18:36:35.725999088 +0000 UTC m=+38.906021207"
Oct 30 18:36:35.732357 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.732328    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:35.841250 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.841197    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fswdh" podStartSLOduration=403.336260407 podStartE2EDuration="6m49.841182536s" totalImagesPullingTime="6.504922129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:35.840466603 +0000 UTC m=+39.020488723" watchObservedRunningTime="2026-10-30 18:36:35.841182536 +0000 UTC m=+39.021204657"
Oct 30 18:36:35.917727 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:35.917631    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:36.121659 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:36.121626    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:36.121659 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:36.121662    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:36.121886 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:36.121838    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:36.121939 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:36.121894    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls podName:a11d470f-d198-42ec-bee1-a900c4814473 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:37.121880008 +0000 UTC m=+40.301902111 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xcwnk" (UID: "a11d470f-d198-42ec-bee1-a900c4814473") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:36.122324 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:36.122298    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:36.122420 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:36.122359    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm podName:a11d470f-d198-42ec-bee1-a900c4814473 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:37.122343963 +0000 UTC m=+40.302366077 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xcwnk" (UID: "a11d470f-d198-42ec-bee1-a900c4814473") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:36.381511 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:36.381478    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:36.876956 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:36.876924    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:37.058002 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:37.057909    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:37.130464 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:37.130387    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:37.130464 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:37.130421    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:37.130687 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:37.130529    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:37.130687 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:37.130607    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm podName:a11d470f-d198-42ec-bee1-a900c4814473 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:39.130586276 +0000 UTC m=+42.310608378 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xcwnk" (UID: "a11d470f-d198-42ec-bee1-a900c4814473") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:37.130687 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:37.130615    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:37.130802 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:37.130697    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls podName:a11d470f-d198-42ec-bee1-a900c4814473 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:39.13067852 +0000 UTC m=+42.310700637 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xcwnk" (UID: "a11d470f-d198-42ec-bee1-a900c4814473") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:37.572332 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:37.571714    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dfv27" event={"ID":"9d148c95-78ae-43af-8942-460853970292","Type":"ContainerStarted","Data":"c4bf75b3847377e194e6d5b828a76d22e206676d9110b81891757659e44d04b0"}
Oct 30 18:36:37.607178 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:37.607145    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:37.607371 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:37.607210    2572 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:37.607612 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:37.607595    2572 scope.go:122] "RemoveContainer" containerID="e7bca663a98eaf8619e4d44e2ab3f7281662e2728ad5d0f643a4a85b0146af10"
Oct 30 18:36:37.607822 ip-10-0-131-224 kubenswrapper[2572]: E1030 18:36:37.607800    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jkgcq_openshift-console-operator(60e0e397-bb2c-4637-9376-c5af871666cb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq" podUID="60e0e397-bb2c-4637-9376-c5af871666cb"
Oct 30 18:36:37.691704 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:37.691643    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dfv27" podStartSLOduration=1.5906430459999998 podStartE2EDuration="3.691625216s" totalImagesPullingTime="2.10098217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:37.690396028 +0000 UTC m=+40.870418150" watchObservedRunningTime="2026-10-30 18:36:37.691625216 +0000 UTC m=+40.871647338"
Oct 30 18:36:37.696817 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:37.696789    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:38.576601 ip-10-0-131-224 kubenswrapper[2572]: I1030 18:36:38.576551    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vs8lx" event={"ID":"92f38b64-26fb-42d6-826a-94cd37e8b668","Type":"ContainerStarted","Data":"6a7b1f5bd35bc7e3c1b4c400acf08f6642bf1404e6da62d01f98cc31e04245b8"}
Oct 30 18:36:38.641982 ip-10-0-131-224 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 18:36:38.662464 ip-10-0-131-224 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 18:36:38.662647 ip-10-0-131-224 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 18:36:38.662833 ip-10-0-131-224 systemd[1]: kubelet.service: Consumed 3.017s CPU time.
Oct 30 18:36:38.676667 ip-10-0-131-224 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 18:36:38.752510 ip-10-0-131-224 kubenswrapper[4924]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:38.752510 ip-10-0-131-224 kubenswrapper[4924]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 18:36:38.752510 ip-10-0-131-224 kubenswrapper[4924]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:38.752510 ip-10-0-131-224 kubenswrapper[4924]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:38.754576 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754488    4924 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:38.754576 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754576    4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:38.754576 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754580    4924 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:38.754673 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754584    4924 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:38.754673 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754587    4924 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:38.754673 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754592    4924 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:38.754673 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754596    4924 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:38.754673 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754600    4924 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:38.754673 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754603    4924 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:38.754673 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754606    4924 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:38.754673 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754609    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:38.754673 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754611    4924 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:38.754673 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754614    4924 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:38.754673 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754626    4924 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:38.754673 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754630    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:38.754673 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754633    4924 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:38.754673 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754636    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:38.754673 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754639    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:38.754673 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754642    4924 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:38.754673 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754645    4924 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:38.754673 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754648    4924 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:38.754673 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754650    4924 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:38.755184 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754653    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:38.755184 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754656    4924 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:38.755184 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754658    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:38.755184 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754661    4924 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:38.755184 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754664    4924 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:38.755184 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754667    4924 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:38.755184 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754670    4924 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:38.755184 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754672    4924 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:38.755184 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754675    4924 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:38.755184 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754678    4924 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:38.755184 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754681    4924 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:38.755184 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754684    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:38.755184 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754686    4924 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:38.755184 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754690    4924 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:38.755184 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754693    4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:38.755184 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754695    4924 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:38.755184 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754698    4924 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:38.755184 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754701    4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:38.755184 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754704    4924 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:38.755662 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754706    4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:38.755662 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754709    4924 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:38.755662 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754712    4924 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:38.755662 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754715    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:38.755662 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754717    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:38.755662 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754720    4924 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:38.755662 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754723    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:38.755662 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754726    4924 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:38.755662 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754728    4924 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:38.755662 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754731    4924 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:38.755662 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754734    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:38.755662 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754736    4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:38.755662 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754739    4924 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:38.755662 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754741    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:38.755662 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754744    4924 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:38.755662 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754747    4924 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:38.755662 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754750    4924 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:38.755662 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754753    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:38.755662 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754756    4924 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:38.755662 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754759    4924 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:38.756168 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754761    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:38.756168 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754764    4924 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:38.756168 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754767    4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:38.756168 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754769    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:38.756168 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754772    4924 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:38.756168 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754774    4924 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:38.756168 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754777    4924 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:38.756168 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754780    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:38.756168 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754782    4924 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:38.756168 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754785    4924 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:38.756168 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754788    4924 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:38.756168 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754790    4924 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:38.756168 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754793    4924 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:38.756168 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754796    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:38.756168 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754799    4924 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:38.756168 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754801    4924 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:38.756168 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754804    4924 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:38.756168 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754806    4924 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:38.756168 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754809    4924 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:38.756643 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754813    4924 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:38.756643 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754815    4924 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:38.756643 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754818    4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:38.756643 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754820    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:38.756643 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754823    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:38.756643 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754826    4924 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:38.756643 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754828    4924 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:38.756643 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754831    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:38.756643 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754834    4924 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:38.756643 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754836    4924 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:38.756643 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754839    4924 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:38.756643 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754842    4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:38.756643 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754844    4924 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:38.756643 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754847    4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:38.756643 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754849    4924 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:38.756643 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754852    4924 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:38.756643 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754855    4924 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:38.756643 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754857    4924 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:38.756643 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754860    4924 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:38.756643 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754862    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:38.757197 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754865    4924 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:38.757197 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754868    4924 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:38.757197 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754870    4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:38.757197 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754873    4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:38.757197 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754875    4924 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:38.757197 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.754878    4924 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:38.757197 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755911    4924 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:38.757197 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755921    4924 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:38.757197 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755925    4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:38.757197 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755928    4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:38.757197 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755931    4924 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:38.757197 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755934    4924 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:38.757197 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755937    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:38.757197 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755940    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:38.757197 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755943    4924 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:38.757197 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755945    4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:38.757197 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755948    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:38.757197 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755952    4924 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:38.757197 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755954    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:38.757730 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755957    4924 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:38.757730 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755960    4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:38.757730 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755963    4924 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:38.757730 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755965    4924 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:38.757730 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755968    4924 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:38.757730 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755970    4924 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:38.757730 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755973    4924 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:38.757730 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755976    4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:38.757730 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755978    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:38.757730 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755981    4924 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:38.757730 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755983    4924 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:38.757730 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755986    4924 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:38.757730 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755988    4924 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:38.757730 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755991    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:38.757730 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.755999    4924 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:38.757730 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756002    4924 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:38.757730 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756005    4924 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:38.757730 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756007    4924 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:38.757730 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756010    4924 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:38.758243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756013    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:38.758243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756015    4924 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:38.758243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756018    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:38.758243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756021    4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:38.758243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756023    4924 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:38.758243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756026    4924 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:38.758243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756029    4924 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:38.758243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756032    4924 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:38.758243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756035    4924 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:38.758243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756038    4924 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:38.758243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756040    4924 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:38.758243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756043    4924 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:38.758243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756046    4924 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:38.758243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756048    4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:38.758243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756051    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:38.758243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756054    4924 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:38.758243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756056    4924 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:38.758243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756059    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:38.758243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756076    4924 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:38.758243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756079    4924 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:38.758770 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756081    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:38.758770 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756084    4924 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:38.758770 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756087    4924 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:38.758770 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756089    4924 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:38.758770 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756092    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:38.758770 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756095    4924 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:38.758770 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756098    4924 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:38.758770 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756101    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:38.758770 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756104    4924 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:38.758770 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756107    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:38.758770 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756110    4924 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:38.758770 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756113    4924 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:38.758770 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756115    4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:38.758770 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756118    4924 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:38.758770 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756120    4924 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:38.758770 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756123    4924 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:38.758770 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756125    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:38.758770 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756128    4924 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:38.758770 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756130    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:38.758770 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756133    4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:38.759277 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756137    4924 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:38.759277 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756140    4924 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:38.759277 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756142    4924 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:38.759277 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756145    4924 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:38.759277 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756148    4924 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:38.759277 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756150    4924 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:38.759277 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756153    4924 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:38.759277 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756156    4924 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:38.759277 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756158    4924 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:38.759277 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756161    4924 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:38.759277 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756165    4924 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:38.759277 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756169    4924 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:38.759277 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756172    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:38.759277 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756175    4924 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:38.759277 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756178    4924 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:38.759277 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756181    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:38.759277 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756184    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:38.759277 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756187    4924 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:38.759277 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756190    4924 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:38.759277 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756193    4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:38.759768 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756196    4924 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:38.759768 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756199    4924 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:38.759768 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756201    4924 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:38.759768 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756204    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:38.759768 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756206    4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:38.759768 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756209    4924 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:38.759768 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756211    4924 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:38.759768 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756214    4924 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:38.759768 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756216    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:38.759768 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756219    4924 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:38.759768 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756221    4924 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:38.759768 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756224    4924 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:38.759768 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756227    4924 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:38.759768 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756230    4924 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:38.759768 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756748    4924 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:38.759768 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756754    4924 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:38.759768 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756759    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:38.759768 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756762    4924 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:38.759768 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756765    4924 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:38.760257 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756770    4924 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:38.760257 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756774    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:38.760257 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756777    4924 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:38.760257 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756780    4924 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:38.760257 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756783    4924 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:38.760257 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756787    4924 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:38.760257 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756790    4924 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:38.760257 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756792    4924 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:38.760257 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756795    4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:38.760257 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756798    4924 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:38.760257 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756801    4924 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:38.760257 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756804    4924 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:38.760257 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756806    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:38.760257 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756809    4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:38.760257 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756811    4924 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:38.760257 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756814    4924 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:38.760257 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756817    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:38.760257 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756819    4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:38.760257 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756822    4924 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:38.760733 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756825    4924 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:38.760733 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756827    4924 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:38.760733 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756830    4924 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:38.760733 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756832    4924 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:38.760733 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756835    4924 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:38.760733 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756837    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:38.760733 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756840    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:38.760733 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756843    4924 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:38.760733 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756846    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:38.760733 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756849    4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:38.760733 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756851    4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:38.760733 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756854    4924 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:38.760733 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756857    4924 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:38.760733 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756859    4924 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:38.760733 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756862    4924 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:38.760733 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756864    4924 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:38.760733 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756867    4924 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:38.760733 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756870    4924 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:38.760733 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756873    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:38.760733 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756875    4924 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:38.761243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756878    4924 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:38.761243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756880    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:38.761243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756883    4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:38.761243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756885    4924 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:38.761243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756888    4924 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:38.761243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756890    4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:38.761243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756893    4924 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:38.761243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756896    4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:38.761243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756898    4924 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:38.761243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756900    4924 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:38.761243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756903    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:38.761243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756905    4924 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:38.761243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756908    4924 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:38.761243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756910    4924 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:38.761243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756913    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:38.761243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756915    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:38.761243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756918    4924 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:38.761243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756920    4924 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:38.761243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756923    4924 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:38.761243 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756925    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:38.761736 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756929    4924 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:38.761736 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756932    4924 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:38.761736 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756934    4924 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:38.761736 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756937    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:38.761736 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756939    4924 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:38.761736 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756942    4924 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:38.761736 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756944    4924 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:38.761736 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756947    4924 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:38.761736 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756949    4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:38.761736 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756953    4924 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:38.761736 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756957    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:38.761736 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756960    4924 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:38.761736 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756963    4924 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:38.761736 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756967    4924 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:38.761736 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756970    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:38.761736 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756972    4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:38.761736 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756975    4924 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:38.761736 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756977    4924 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:38.761736 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756980    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:38.762228 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756984    4924 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:38.762228 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756987    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:38.762228 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756989    4924 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:38.762228 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756992    4924 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:38.762228 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756995    4924 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:38.762228 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.756997    4924 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:38.762228 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757000    4924 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:38.762228 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757002    4924 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:38.762228 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757005    4924 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:38.762228 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757007    4924 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:38.762228 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757009    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:38.762228 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757012    4924 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:38.762228 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757015    4924 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:38.762228 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757017    4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:38.762228 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757020    4924 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:38.762228 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757023    4924 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:38.762228 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757026    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:38.762228 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757028    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:38.762228 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757031    4924 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:38.762228 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757033    4924 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:38.762725 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757037    4924 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:38.762725 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757039    4924 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:38.762725 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757042    4924 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:38.762725 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757158    4924 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 18:36:38.762725 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757167    4924 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 18:36:38.762725 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757174    4924 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 18:36:38.762725 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757179    4924 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 18:36:38.762725 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757183    4924 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 18:36:38.762725 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757187    4924 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 18:36:38.762725 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757192    4924 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 18:36:38.762725 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757196    4924 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 18:36:38.762725 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757199    4924 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 18:36:38.762725 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757204    4924 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 18:36:38.762725 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757208    4924 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 18:36:38.762725 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757211    4924 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 18:36:38.762725 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757215    4924 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 18:36:38.762725 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757218    4924 flags.go:64] FLAG: --cgroup-root=""
Oct 30 18:36:38.762725 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757221    4924 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 18:36:38.762725 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757224    4924 flags.go:64] FLAG: --client-ca-file=""
Oct 30 18:36:38.762725 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757227    4924 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 18:36:38.762725 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757230    4924 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 18:36:38.762725 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757235    4924 flags.go:64] FLAG: --cluster-domain=""
Oct 30 18:36:38.762725 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757237    4924 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 18:36:38.762725 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757240    4924 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 18:36:38.763356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757244    4924 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 18:36:38.763356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757247    4924 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 18:36:38.763356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757251    4924 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 18:36:38.763356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757255    4924 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 18:36:38.763356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757258    4924 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 18:36:38.763356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757265    4924 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 18:36:38.763356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757268    4924 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 18:36:38.763356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757272    4924 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 18:36:38.763356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757274    4924 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 18:36:38.763356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757278    4924 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 18:36:38.763356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757281    4924 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 18:36:38.763356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757285    4924 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 18:36:38.763356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757288    4924 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 18:36:38.763356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757291    4924 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 18:36:38.763356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757294    4924 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 18:36:38.763356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757297    4924 flags.go:64] FLAG: --enable-server="true"
Oct 30 18:36:38.763356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757300    4924 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 18:36:38.763356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757304    4924 flags.go:64] FLAG: --event-burst="100"
Oct 30 18:36:38.763356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757308    4924 flags.go:64] FLAG: --event-qps="50"
Oct 30 18:36:38.763356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757311    4924 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 18:36:38.763356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757315    4924 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 18:36:38.763356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757318    4924 flags.go:64] FLAG: --eviction-hard=""
Oct 30 18:36:38.763356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757323    4924 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 18:36:38.763356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757326    4924 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 18:36:38.763928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757329    4924 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 18:36:38.763928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757332    4924 flags.go:64] FLAG: --eviction-soft=""
Oct 30 18:36:38.763928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757335    4924 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 18:36:38.763928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757338    4924 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 18:36:38.763928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757342    4924 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 18:36:38.763928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757345    4924 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 18:36:38.763928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757347    4924 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 18:36:38.763928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757350    4924 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 18:36:38.763928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757353    4924 flags.go:64] FLAG: --feature-gates=""
Oct 30 18:36:38.763928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757360    4924 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 18:36:38.763928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757363    4924 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 18:36:38.763928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757367    4924 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 18:36:38.763928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757370    4924 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 18:36:38.763928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757376    4924 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 18:36:38.763928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757379    4924 flags.go:64] FLAG: --help="false"
Oct 30 18:36:38.763928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757382    4924 flags.go:64] FLAG: --hostname-override="ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:36:38.763928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757385    4924 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 18:36:38.763928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757395    4924 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 18:36:38.763928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757399    4924 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 18:36:38.763928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757402    4924 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 18:36:38.763928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757406    4924 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 18:36:38.763928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757410    4924 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 18:36:38.763928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757413    4924 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 18:36:38.764551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757415    4924 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 18:36:38.764551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757418    4924 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 18:36:38.764551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757421    4924 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 18:36:38.764551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757424    4924 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 18:36:38.764551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757427    4924 flags.go:64] FLAG: --kube-reserved=""
Oct 30 18:36:38.764551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757431    4924 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 18:36:38.764551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757435    4924 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 18:36:38.764551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757438    4924 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 18:36:38.764551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757441    4924 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 18:36:38.764551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757444    4924 flags.go:64] FLAG: --lock-file=""
Oct 30 18:36:38.764551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757447    4924 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 18:36:38.764551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757450    4924 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 18:36:38.764551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757453    4924 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 18:36:38.764551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757458    4924 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 18:36:38.764551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757461    4924 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 18:36:38.764551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757464    4924 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 18:36:38.764551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757467    4924 flags.go:64] FLAG: --logging-format="text"
Oct 30 18:36:38.764551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757470    4924 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 18:36:38.764551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757473    4924 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 18:36:38.764551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757476    4924 flags.go:64] FLAG: --manifest-url=""
Oct 30 18:36:38.764551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757480    4924 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 18:36:38.764551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757484    4924 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 18:36:38.764551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757489    4924 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 18:36:38.764551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757493    4924 flags.go:64] FLAG: --max-pods="110"
Oct 30 18:36:38.764551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757496    4924 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 18:36:38.765183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757499    4924 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 18:36:38.765183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757502    4924 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 18:36:38.765183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757505    4924 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 18:36:38.765183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757509    4924 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 18:36:38.765183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757511    4924 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 18:36:38.765183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757515    4924 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 18:36:38.765183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757522    4924 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 18:36:38.765183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757525    4924 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 18:36:38.765183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757528    4924 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 18:36:38.765183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757531    4924 flags.go:64] FLAG: --pod-cidr=""
Oct 30 18:36:38.765183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757534    4924 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 18:36:38.765183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757536    4924 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 18:36:38.765183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757541    4924 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 18:36:38.765183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757544    4924 flags.go:64] FLAG: --port="10250"
Oct 30 18:36:38.765183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757547    4924 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 18:36:38.765183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757550    4924 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0d1e1663a488cdaa6"
Oct 30 18:36:38.765183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757553    4924 flags.go:64] FLAG: --qos-reserved=""
Oct 30 18:36:38.765183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757556    4924 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 18:36:38.765183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757559    4924 flags.go:64] FLAG: --register-node="true"
Oct 30 18:36:38.765183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757562    4924 flags.go:64] FLAG: --register-with-taints=""
Oct 30 18:36:38.765183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757565    4924 flags.go:64] FLAG: --registry-burst="10"
Oct 30 18:36:38.765183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757568    4924 flags.go:64] FLAG: --registry-qps="5"
Oct 30 18:36:38.765183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757571    4924 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 18:36:38.765183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757574    4924 flags.go:64] FLAG: --reserved-memory=""
Oct 30 18:36:38.765183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757578    4924 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 18:36:38.765183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757581    4924 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 18:36:38.766089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757584    4924 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 18:36:38.766089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757587    4924 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 18:36:38.766089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757590    4924 flags.go:64] FLAG: --runonce="false"
Oct 30 18:36:38.766089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757594    4924 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 18:36:38.766089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757598    4924 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 18:36:38.766089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757601    4924 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 18:36:38.766089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757604    4924 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 18:36:38.766089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757607    4924 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 18:36:38.766089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757611    4924 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 18:36:38.766089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757614    4924 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 18:36:38.766089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757617    4924 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 18:36:38.766089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757620    4924 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 18:36:38.766089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757623    4924 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 18:36:38.766089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757626    4924 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 18:36:38.766089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757629    4924 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 18:36:38.766089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757632    4924 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 18:36:38.766089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757635    4924 flags.go:64] FLAG: --system-cgroups=""
Oct 30 18:36:38.766089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757638    4924 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 18:36:38.766089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757644    4924 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 18:36:38.766089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757648    4924 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 18:36:38.766089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757650    4924 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 18:36:38.766089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757655    4924 flags.go:64] FLAG: --tls-min-version=""
Oct 30 18:36:38.766089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757658    4924 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 18:36:38.766089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757661    4924 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 18:36:38.766089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757664    4924 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 18:36:38.766800 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757667    4924 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 18:36:38.766800 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757670    4924 flags.go:64] FLAG: --v="2"
Oct 30 18:36:38.766800 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757674    4924 flags.go:64] FLAG: --version="false"
Oct 30 18:36:38.766800 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757678    4924 flags.go:64] FLAG: --vmodule=""
Oct 30 18:36:38.766800 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757682    4924 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 18:36:38.766800 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.757686    4924 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 18:36:38.766800 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757820    4924 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:38.766800 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757823    4924 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:38.766800 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757827    4924 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:38.766800 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757830    4924 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:38.766800 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757834    4924 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:38.766800 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757837    4924 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:38.766800 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757840    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:38.766800 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757842    4924 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:38.766800 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757845    4924 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:38.766800 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757848    4924 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:38.766800 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757851    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:38.766800 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757853    4924 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:38.766800 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757856    4924 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:38.766800 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757859    4924 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:38.766800 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757861    4924 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:38.767345 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757864    4924 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:38.767345 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757867    4924 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:38.767345 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757869    4924 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:38.767345 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757872    4924 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:38.767345 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757876    4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:38.767345 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757879    4924 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:38.767345 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757881    4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:38.767345 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757884    4924 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:38.767345 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757887    4924 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:38.767345 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757889    4924 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:38.767345 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757892    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:38.767345 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757895    4924 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:38.767345 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757897    4924 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:38.767345 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757900    4924 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:38.767345 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757902    4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:38.767345 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757905    4924 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:38.767345 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757908    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:38.767345 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757910    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:38.767345 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757913    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:38.767345 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757915    4924 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:38.767854 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757918    4924 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:38.767854 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757923    4924 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:38.767854 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757926    4924 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:38.767854 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757930    4924 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:38.767854 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757934    4924 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:38.767854 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757938    4924 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:38.767854 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757942    4924 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:38.767854 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757945    4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:38.767854 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757947    4924 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:38.767854 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757951    4924 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:38.767854 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757955    4924 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:38.767854 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757958    4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:38.767854 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757961    4924 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:38.767854 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757963    4924 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:38.767854 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757966    4924 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:38.767854 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757969    4924 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:38.767854 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757972    4924 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:38.767854 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757976    4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:38.767854 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757978    4924 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:38.768402 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757981    4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:38.768402 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757984    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:38.768402 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757986    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:38.768402 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757989    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:38.768402 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757991    4924 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:38.768402 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757994    4924 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:38.768402 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757996    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:38.768402 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.757999    4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:38.768402 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758001    4924 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:38.768402 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758004    4924 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:38.768402 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758007    4924 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:38.768402 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758009    4924 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:38.768402 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758012    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:38.768402 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758014    4924 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:38.768402 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758018    4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:38.768402 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758020    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:38.768402 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758023    4924 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:38.768402 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758026    4924 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:38.768402 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758029    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:38.768940 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758031    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:38.768940 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758034    4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:38.768940 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758036    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:38.768940 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758039    4924 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:38.768940 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758041    4924 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:38.768940 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758044    4924 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:38.768940 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758046    4924 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:38.768940 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758049    4924 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:38.768940 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758051    4924 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:38.768940 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758054    4924 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:38.768940 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758057    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:38.768940 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758074    4924 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:38.768940 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758079    4924 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:38.768940 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758081    4924 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:38.768940 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758084    4924 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:38.768940 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758087    4924 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:38.768940 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758090    4924 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:38.768940 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758092    4924 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:38.768940 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758095    4924 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:38.768940 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758097    4924 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:38.769677 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758100    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:38.769677 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758103    4924 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:38.769677 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758105    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:38.769677 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758107    4924 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:38.769677 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758110    4924 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:38.769677 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758113    4924 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:38.769677 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758115    4924 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:38.769677 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758119    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:38.769677 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758122    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:38.769677 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758124    4924 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:38.769677 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758127    4924 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:38.769677 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758130    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:38.769677 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758133    4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:38.769677 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.758160    4924 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:38.769677 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758197    4924 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:38.769677 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758202    4924 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:38.770280 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758205    4924 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:38.770280 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758208    4924 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:38.770280 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758213    4924 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:38.770280 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758216    4924 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:38.770280 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758219    4924 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:38.770280 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758222    4924 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:38.770280 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758227    4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:38.770280 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758230    4924 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:38.770280 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758234    4924 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:38.770280 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758238    4924 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:38.770280 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758241    4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:38.770280 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758244    4924 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:38.770280 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758246    4924 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:38.770280 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758249    4924 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:38.770280 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758252    4924 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:38.770280 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758255    4924 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:38.770280 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758257    4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:38.770280 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758260    4924 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:38.770280 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758263    4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:38.770781 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758265    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:38.770781 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758268    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:38.770781 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758271    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:38.770781 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758274    4924 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:38.770781 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758276    4924 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:38.770781 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758280    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:38.770781 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758283    4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:38.770781 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758285    4924 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:38.770781 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758288    4924 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:38.770781 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758291    4924 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:38.770781 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758294    4924 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:38.770781 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758297    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:38.770781 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758299    4924 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:38.770781 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758302    4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:38.770781 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758305    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:38.770781 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758307    4924 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:38.770781 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758310    4924 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:38.770781 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758313    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:38.771434 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758315    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:38.771434 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758320    4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:38.771434 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758322    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:38.771434 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758325    4924 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:38.771434 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758328    4924 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:38.771434 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758330    4924 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:38.771434 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758333    4924 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:38.771434 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758336    4924 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:38.771434 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758338    4924 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:38.771434 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758341    4924 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:38.771434 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758344    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:38.771434 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758346    4924 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:38.771434 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758349    4924 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:38.771434 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758351    4924 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:38.771434 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758354    4924 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:38.771434 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758356    4924 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:38.771434 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758359    4924 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:38.771434 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758362    4924 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:38.771434 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758364    4924 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:38.771434 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758368    4924 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:38.772077 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758371    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:38.772077 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758373    4924 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:38.772077 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758376    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:38.772077 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758378    4924 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:38.772077 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758381    4924 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:38.772077 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758384    4924 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:38.772077 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758386    4924 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:38.772077 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758389    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:38.772077 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758392    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:38.772077 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758395    4924 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:38.772077 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758397    4924 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:38.772077 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758399    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:38.772077 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758402    4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:38.772077 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758406    4924 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:38.772077 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758408    4924 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:38.772077 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758412    4924 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:38.772077 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758415    4924 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:38.772077 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758418    4924 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:38.772077 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758420    4924 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:38.772077 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758423    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:38.772667 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758425    4924 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:38.772667 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758428    4924 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:38.772667 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758431    4924 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:38.772667 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758434    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:38.772667 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758436    4924 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:38.772667 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758439    4924 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:38.772667 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758442    4924 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:38.772667 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758444    4924 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:38.772667 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758447    4924 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:38.772667 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758449    4924 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:38.772667 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758452    4924 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:38.772667 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758456    4924 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:38.772667 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758459    4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:38.772667 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758461    4924 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:38.772667 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758464    4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:38.772667 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758467    4924 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:38.772667 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758469    4924 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:38.772667 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758472    4924 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:38.772667 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758478    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:38.772667 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758481    4924 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:38.773249 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758483    4924 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:38.773249 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758486    4924 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:38.773249 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758489    4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:38.773249 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758492    4924 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:38.773249 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758495    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:38.773249 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758498    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:38.773249 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.758501    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:38.773249 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.762040    4924 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 18:36:38.773249 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.762054    4924 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 18:36:38.773249 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.762083    4924 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 18:36:38.773249 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.762089    4924 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 18:36:38.773249 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762144    4924 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:38.773249 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762149    4924 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:38.773249 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762153    4924 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:38.773249 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762156    4924 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:38.773249 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762159    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:38.773249 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762162    4924 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:38.773249 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762165    4924 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:38.773249 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762167    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:38.773249 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762170    4924 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:38.773745 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762173    4924 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:38.773745 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762177    4924 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:38.773745 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762181    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:38.773745 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762184    4924 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:38.773745 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762187    4924 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:38.773745 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762190    4924 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:38.773745 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762193    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:38.773745 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762195    4924 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:38.773745 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762198    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:38.773745 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762202    4924 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:38.773745 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762209    4924 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:38.773745 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762212    4924 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:38.773745 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762215    4924 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:38.773745 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762218    4924 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:38.773745 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762221    4924 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:38.773745 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762223    4924 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:38.773745 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762226    4924 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:38.773745 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762229    4924 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:38.773745 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762232    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:38.774223 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762234    4924 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:38.774223 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762237    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:38.774223 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762240    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:38.774223 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762242    4924 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:38.774223 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762245    4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:38.774223 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762247    4924 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:38.774223 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762250    4924 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:38.774223 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762252    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:38.774223 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762255    4924 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:38.774223 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762258    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:38.774223 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762260    4924 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:38.774223 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762263    4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:38.774223 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762265    4924 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:38.774223 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762268    4924 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:38.774223 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762270    4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:38.774223 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762273    4924 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:38.774223 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762275    4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:38.774223 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762277    4924 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:38.774223 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762280    4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:38.774223 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762283    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:38.774713 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762285    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:38.774713 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762287    4924 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:38.774713 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762292    4924 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:38.774713 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762294    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:38.774713 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762298    4924 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:38.774713 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762300    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:38.774713 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762303    4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:38.774713 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762305    4924 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:38.774713 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762308    4924 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:38.774713 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762311    4924 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:38.774713 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762313    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:38.774713 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762315    4924 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:38.774713 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762318    4924 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:38.774713 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762320    4924 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:38.774713 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762323    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:38.774713 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762325    4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:38.774713 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762327    4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:38.774713 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762330    4924 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:38.774713 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762332    4924 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:38.774713 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762335    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:38.775212 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762337    4924 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:38.775212 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762340    4924 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:38.775212 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762342    4924 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:38.775212 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762345    4924 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:38.775212 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762347    4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:38.775212 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762350    4924 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:38.775212 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762352    4924 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:38.775212 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762355    4924 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:38.775212 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762357    4924 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:38.775212 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762360    4924 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:38.775212 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762362    4924 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:38.775212 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762365    4924 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:38.775212 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762367    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:38.775212 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762370    4924 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:38.775212 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762373    4924 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:38.775212 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762376    4924 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:38.775212 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762378    4924 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:38.775212 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762381    4924 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:38.775212 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762384    4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:38.775212 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762387    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:38.775695 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762389    4924 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:38.775695 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762391    4924 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:38.775695 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762394    4924 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:38.775695 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762396    4924 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:38.775695 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762399    4924 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:38.775695 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762401    4924 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:38.775695 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762404    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:38.775695 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762406    4924 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:38.775695 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762409    4924 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:38.775695 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762411    4924 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:38.775695 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762414    4924 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:38.775695 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762418    4924 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:38.775695 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762421    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:38.775695 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762424    4924 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:38.775695 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762426    4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:38.775695 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762429    4924 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:38.775695 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762432    4924 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:38.775695 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762434    4924 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:38.776167 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.762460    4924 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:38.776167 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762592    4924 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:38.776167 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762596    4924 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:38.776167 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762599    4924 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:38.776167 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762602    4924 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:38.776167 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762605    4924 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:38.776167 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762608    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:38.776167 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762611    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:38.776167 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762614    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:38.776167 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762616    4924 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:38.776167 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762618    4924 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:38.776167 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762621    4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:38.776167 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762624    4924 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:38.776167 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762627    4924 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:38.776167 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762629    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:38.776534 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762632    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:38.776534 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762634    4924 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:38.776534 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762637    4924 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:38.776534 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762640    4924 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:38.776534 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762642    4924 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:38.776534 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762645    4924 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:38.776534 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762647    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:38.776534 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762650    4924 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:38.776534 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762652    4924 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:38.776534 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762655    4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:38.776534 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762657    4924 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:38.776534 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762659    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:38.776534 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762662    4924 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:38.776534 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762665    4924 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:38.776534 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762667    4924 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:38.776534 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762670    4924 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:38.776534 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762672    4924 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:38.776534 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762675    4924 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:38.776534 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762677    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:38.777027 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762680    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:38.777027 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762682    4924 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:38.777027 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762684    4924 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:38.777027 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762687    4924 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:38.777027 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762689    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:38.777027 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762692    4924 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:38.777027 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762695    4924 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:38.777027 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762698    4924 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:38.777027 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762700    4924 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:38.777027 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762703    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:38.777027 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762705    4924 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:38.777027 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762708    4924 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:38.777027 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762728    4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:38.777027 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762732    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:38.777027 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762736    4924 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:38.777027 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762739    4924 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:38.777027 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762743    4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:38.777027 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762745    4924 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:38.777027 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762748    4924 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:38.777027 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762751    4924 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:38.777544 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762755    4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:38.777544 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762757    4924 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:38.777544 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762760    4924 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:38.777544 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762763    4924 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:38.777544 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762765    4924 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:38.777544 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762768    4924 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:38.777544 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762771    4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:38.777544 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762773    4924 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:38.777544 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762776    4924 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:38.777544 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762778    4924 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:38.777544 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762781    4924 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:38.777544 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762783    4924 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:38.777544 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762786    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:38.777544 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762788    4924 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:38.777544 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762791    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:38.777544 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762793    4924 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:38.777544 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762795    4924 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:38.777544 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762798    4924 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:38.777544 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762801    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:38.777544 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762804    4924 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:38.778032 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762807    4924 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:38.778032 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762809    4924 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:38.778032 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762812    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:38.778032 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762814    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:38.778032 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762817    4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:38.778032 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762820    4924 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:38.778032 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762822    4924 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:38.778032 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762825    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:38.778032 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762827    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:38.778032 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762830    4924 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:38.778032 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762833    4924 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:38.778032 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762836    4924 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:38.778032 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762838    4924 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:38.778032 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762841    4924 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:38.778032 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762843    4924 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:38.778032 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762846    4924 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:38.778032 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762848    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:38.778032 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762852    4924 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:38.778032 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762855    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:38.778687 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762858    4924 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:38.778687 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762860    4924 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:38.778687 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762863    4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:38.778687 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762865    4924 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:38.778687 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762868    4924 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:38.778687 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762871    4924 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:38.778687 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762873    4924 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:38.778687 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762875    4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:38.778687 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762878    4924 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:38.778687 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762881    4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:38.778687 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762883    4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:38.778687 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762886    4924 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:38.778687 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762888    4924 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:38.778687 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762892    4924 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:38.778687 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.762914    4924 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:38.779094 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762947    4924 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:38.779094 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762950    4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:38.779094 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762953    4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:38.779094 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762956    4924 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:38.779094 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762959    4924 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:38.779094 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762962    4924 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:38.779094 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762966    4924 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:38.779094 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762969    4924 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:38.779094 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762972    4924 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:38.779094 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762975    4924 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:38.779094 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762977    4924 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:38.779094 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762980    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:38.779094 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762983    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:38.779094 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762985    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:38.779094 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762988    4924 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:38.779094 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762990    4924 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:38.779094 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762993    4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:38.779094 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762996    4924 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:38.779540 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.762998    4924 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:38.779540 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763001    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:38.779540 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763003    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:38.779540 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763006    4924 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:38.779540 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763008    4924 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:38.779540 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763011    4924 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:38.779540 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763013    4924 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:38.779540 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763016    4924 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:38.779540 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763018    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:38.779540 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763021    4924 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:38.779540 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763023    4924 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:38.779540 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763029    4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:38.779540 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763031    4924 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:38.779540 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763034    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:38.779540 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763040    4924 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:38.779540 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763043    4924 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:38.779540 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763045    4924 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:38.779540 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763048    4924 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:38.779540 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763051    4924 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:38.779540 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763053    4924 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:38.780038 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763056    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:38.780038 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763058    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:38.780038 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763072    4924 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:38.780038 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763075    4924 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:38.780038 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763078    4924 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:38.780038 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763081    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:38.780038 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763083    4924 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:38.780038 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763086    4924 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:38.780038 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763088    4924 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:38.780038 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763091    4924 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:38.780038 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763093    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:38.780038 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763096    4924 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:38.780038 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763099    4924 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:38.780038 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763101    4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:38.780038 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763104    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:38.780038 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763107    4924 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:38.780038 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763109    4924 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:38.780038 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763112    4924 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:38.780038 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763115    4924 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:38.780038 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763117    4924 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:38.780578 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763120    4924 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:38.780578 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763122    4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:38.780578 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763125    4924 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:38.780578 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763129    4924 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:38.780578 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763133    4924 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:38.780578 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763135    4924 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:38.780578 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763139    4924 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:38.780578 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763142    4924 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:38.780578 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763144    4924 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:38.780578 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763147    4924 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:38.780578 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763150    4924 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:38.780578 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763153    4924 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:38.780578 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763156    4924 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:38.780578 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763158    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:38.780578 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763161    4924 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:38.780578 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763163    4924 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:38.780578 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763166    4924 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:38.780578 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763169    4924 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:38.780578 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763171    4924 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:38.780578 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763174    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:38.781087 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763177    4924 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:38.781087 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763179    4924 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:38.781087 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763182    4924 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:38.781087 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763185    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:38.781087 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763187    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:38.781087 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763189    4924 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:38.781087 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763192    4924 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:38.781087 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763195    4924 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:38.781087 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763197    4924 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:38.781087 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763200    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:38.781087 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763202    4924 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:38.781087 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763205    4924 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:38.781087 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763207    4924 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:38.781087 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763210    4924 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:38.781087 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763213    4924 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:38.781087 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763218    4924 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:38.781087 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763221    4924 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:38.781087 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763223    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:38.781087 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763230    4924 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:38.781561 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763233    4924 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:38.781561 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763236    4924 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:38.781561 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763239    4924 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:38.781561 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763241    4924 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:38.781561 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763244    4924 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:38.781561 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763247    4924 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:38.781561 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763249    4924 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:38.781561 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763252    4924 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:38.781561 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:38.763254    4924 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:38.781561 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.763301    4924 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 18:36:38.781561 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.764241    4924 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 18:36:38.781561 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.764296    4924 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 18:36:38.781561 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.764471    4924 server.go:1014] "Starting client certificate rotation"
Oct 30 18:36:38.781561 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.764600    4924 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:36:38.781561 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.764638    4924 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:30:57 +0000 UTC" deadline="2028-04-24 08:25:39.973297793 +0000 UTC"
Oct 30 18:36:38.781561 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.764664    4924 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12997h49m1.208635597s"
Oct 30 18:36:38.781984 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.764874    4924 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:36:38.781984 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.767821    4924 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:36:38.781984 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.769576    4924 log.go:25] "Validated CRI v1 runtime API"
Oct 30 18:36:38.781984 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.771398    4924 log.go:25] "Validated CRI v1 image API"
Oct 30 18:36:38.781984 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.772127    4924 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 18:36:38.781984 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.773965    4924 fs.go:135] Filesystem UUIDs: map[73edc71b-6b7b-4a14-bbaa-705df54b3218:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e43d6a5c-0382-4583-8ce2-efaa9d60efbe:/dev/nvme0n1p4]
Oct 30 18:36:38.782160 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.773982    4924 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/022157ce1cf92e0afd69fe9ec69a54f1aa1da7587493d5ce91ef80846f269ccf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/022157ce1cf92e0afd69fe9ec69a54f1aa1da7587493d5ce91ef80846f269ccf/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/14de059988139725a952bc59f04bd0c84c5561ad20b6de9d807f2577d2cb131a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/14de059988139725a952bc59f04bd0c84c5561ad20b6de9d807f2577d2cb131a/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e099f4ea8091b1a7d81d6f31319a778089bcb24400d3e279d44461982cda02d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e099f4ea8091b1a7d81d6f31319a778089bcb24400d3e279d44461982cda02d/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3aaf547da0a47eca32ce390a3406034d7022e20fea73e56992f72657a781c2dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3aaf547da0a47eca32ce390a3406034d7022e20fea73e56992f72657a781c2dd/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c56149a6c66eaa82146cbbab520741d290fbb1283425fd10aa2409a3f28accf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c56149a6c66eaa82146cbbab520741d290fbb1283425fd10aa2409a3f28accf/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/47de5c3c76d6b584d3f74b82db93dff272d550428be9da41ddfe3619fa3ec2d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/47de5c3c76d6b584d3f74b82db93dff272d550428be9da41ddfe3619fa3ec2d5/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a5a5bcab19b8f9cd4474e125f5e0e25e6671882e7c8e722fbeb629c8f8f3821/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a5a5bcab19b8f9cd4474e125f5e0e25e6671882e7c8e722fbeb629c8f8f3821/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5edaeb3ad7c0c9d0fe1f145eeace4b609b052a02d7b21055e42ca516feff8f11/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5edaeb3ad7c0c9d0fe1f145eeace4b609b052a02d7b21055e42ca516feff8f11/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b7dd88b77a31d2d61979b2e9b046603cc78f3aae97bc95a0bac4ddbf3309384/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b7dd88b77a31d2d61979b2e9b046603cc78f3aae97bc95a0bac4ddbf3309384/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80977e6181a3c742059b0adae7b88dbd4190d264ac3da7c81df5c9a38370a97d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80977e6181a3c742059b0adae7b88dbd4190d264ac3da7c81df5c9a38370a97d/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ecb952775b583a3883cc36ec2b142c98891e7d39ff126a4edcd4a1eef4131f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ecb952775b583a3883cc36ec2b142c98891e7d39ff126a4edcd4a1eef4131f3/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90724e59845a496535c3fa041fad3cb95848b518bd1225c142dd012354e9e99e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90724e59845a496535c3fa041fad3cb95848b518bd1225c142dd012354e9e99e/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a0551ed10d5edfadfa22b5ba6600154ed5ee9ff3090c925802c7867c7e97bfec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a0551ed10d5edfadfa22b5ba6600154ed5ee9ff3090c925802c7867c7e97bfec/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aaffbe47de8f1818df45a736effd1ad88bf5a61518d31a7a73b2695a9f07e400/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aaffbe47de8f1818df45a736effd1ad88bf5a61518d31a7a73b2695a9f07e400/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5e736af7cb22e7e295d2c82ed4c7548f4498a44bbd419920772c5a90e05ce4e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5e736af7cb22e7e295d2c82ed4c7548f4498a44bbd419920772c5a90e05ce4e/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b982014ee3a54d7d3cb4c6619b1f3d0b15954c25fd5bac0f95a58c78a3541556/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b982014ee3a54d7d3cb4c6619b1f3d0b15954c25fd5bac0f95a58c78a3541556/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf414281ada08cae7668511c127437eedad35ede1b4ec220a53de48da3f35ebe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf414281ada08cae7668511c127437eedad35ede1b4ec220a53de48da3f35ebe/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d42e55117f9bd1279beabb43c4a75ceb656285a170dffb6bdc9e7531b4666440/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d42e55117f9bd1279beabb43c4a75ceb656285a170dffb6bdc9e7531b4666440/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e4755db47b4602d74bc92410b4139a0c03ec31fe285b49cc74c99928d8351c06/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e4755db47b4602d74bc92410b4139a0c03ec31fe285b49cc74c99928d8351c06/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f46bb25af1cc8f5165ee09c930e7a01203af203a8515498da17f95919c2cc4f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f46bb25af1cc8f5165ee09c930e7a01203af203a8515498da17f95919c2cc4f4/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f863f9d519de3c1bdafab1e8717dd8731b7fafdd7c9c7b23a372d5d64b18c521/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f863f9d519de3c1bdafab1e8717dd8731b7fafdd7c9c7b23a372d5d64b18c521/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f8e7bbb9d66216640a2e4edc5ffdb3166448ec13bc4800d1523680d155f3634b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f8e7bbb9d66216640a2e4edc5ffdb3166448ec13bc4800d1523680d155f3634b/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9746669de1d07f5c2c427a3f47b253acf5cab3b53fd2d2afe6019626e24c8ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9746669de1d07f5c2c427a3f47b253acf5cab3b53fd2d2afe6019626e24c8ba/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05fe2f04-29f7-4999-87ef-b30035b34883/volumes/kubernetes.io~projected/kube-api-access-mmjgg:{mountpoint:/var/lib/kubelet/pods/05fe2f04-29f7-4999-87ef-b30035b34883/volumes/kubernetes.io~projected/kube-api-access-mmjgg major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05fe2f04-29f7-4999-87ef-b30035b34883/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/05fe2f04-29f7-4999-87ef-b30035b34883/volumes/kubernetes.io~secret/default-certificate major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05fe2f04-29f7-4999-87ef-b30035b34883/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/05fe2f04-29f7-4999-87ef-b30035b34883/volumes/kubernetes.io~secret/stats-auth major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0648f559-00ac-4de0-8b9f-ce02d1b657a9/volumes/kubernetes.io~projected/kube-api-access-k8ggz:{mountpoint:/var/lib/kubelet/pods/0648f559-00ac-4de0-8b9f-ce02d1b657a9/volumes/kubernetes.io~projected/kube-api-access-k8ggz major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0648f559-00ac-4de0-8b9f-ce02d1b657a9/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/0648f559-00ac-4de0-8b9f-ce02d1b657a9/volumes/kubernetes.io~secret/serving-cert major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278/volumes/kubernetes.io~projected/kube-api-access-vb856:{mountpoint:/var/lib/kubelet/pods/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278/volumes/kubernetes.io~projected/kube-api-access-vb856 major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12928ad7-4038-473a-a4aa-be1a6a42eabc/volumes/kubernetes.io~projected/kube-api-access-qcz45:{mountpoint:/var/lib/kubelet/pods/12928ad7-4038-473a-a4aa-be1a6a42eabc/volumes/kubernetes.io~projected/kube-api-access-qcz45 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e0da5dc-3a50-414f-99aa-0a8bbe303a31/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/1e0da5dc-3a50-414f-99aa-0a8bbe303a31/volumes/kubernetes.io~secret/agent-certs major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21b6f672-2dc5-4523-9dfa-6dfaa30a8bac/volumes/kubernetes.io~projected/kube-api-access-ks7sd:{mountpoint:/var/lib/kubelet/pods/21b6f672-2dc5-4523-9dfa-6dfaa30a8bac/volumes/kubernetes.io~projected/kube-api-access-ks7sd major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21b6f672-2dc5-4523-9dfa-6dfaa30a8bac/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/21b6f672-2dc5-4523-9dfa-6dfaa30a8bac/volumes/kubernetes.io~secret/cert major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3935b0fd-69be-4272-adc5-ecfbc7af04ae/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/3935b0fd-69be-4272-adc5-ecfbc7af04ae/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3935b0fd-69be-4272-adc5-ecfbc7af04ae/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/3935b0fd-69be-4272-adc5-ecfbc7af04ae/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3935b0fd-69be-4272-adc5-ecfbc7af04ae/volumes/kubernetes.io~projected/kube-api-access-g6bwn:{mountpoint:/var/lib/kubelet/pods/3935b0fd-69be-4272-adc5-ecfbc7af04ae/volumes/kubernetes.io~projected/kube-api-access-g6bwn major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d05b776-9750-4c5b-8d91-4cf27329a1fc/volumes/kubernetes.io~projected/kube-api-access-2xg8f:{mountpoint:/var/lib/kubelet/pods/5d05b776-9750-4c5b-8d91-4cf27329a1fc/volumes/kubernetes.io~projected/kube-api-access-2xg8f major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ef02004-71fc-4464-9550-bd094d27f21f/volumes/kubernetes.io~projected/kube-api-access-dx8hm:{mountpoint:/var/lib/kubelet/pods/5ef02004-71fc-4464-9550-bd094d27f21f/volumes/kubernetes.io~projected/kube-api-access-dx8hm major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60e0e397-bb2c-4637-9376-c5af871666cb/volumes/kubernetes.io~projected/kube-api-access-v526v:{mountpoint:/var/lib/kubelet/pods/60e0e397-bb2c-4637-9376-c5af871666cb/volumes/kubernetes.io~projected/kube-api-access-v526v major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60e0e397-bb2c-4637-9376-c5af871666cb/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/60e0e397-bb2c-4637-9376-c5af871666cb/volumes/kubernetes.io~secret/serving-cert major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65beb5a5-1df3-42c8-9e14-769b4ad47255/volumes/kubernetes.io~projected/kube-api-access-jpqmj:{mountpoint:/var/lib/kubelet/pods/65beb5a5-1df3-42c8-9e14-769b4ad47255/volumes/kubernetes.io~projected/kube-api-access-jpqmj major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6df0f1e7-07b7-4231-bc70-ba81aef29856/volumes/kubernetes.io~projected/kube-api-access-4fdth:{mountpoint:/var/lib/kubelet/pods/6df0f1e7-07b7-4231-bc70-ba81aef29856/volumes/kubernetes.io~projected/kube-api-access-4fdth major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74a85732-6555-48bf-9ece-4e863cd1fb93/volumes/kubernetes.io~projected/kube-api-access-qkl9j:{mountpoint:/var/lib/kubelet/pods/74a85732-6555-48bf-9ece-4e863cd1fb93/volumes/kubernetes.io~projected/kube-api-access-qkl9j major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80e45018-1682-47cd-9523-dbeaf35a39cd/volumes/kubernetes.io~projected/kube-api-access-vdblb:{mountpoint:/var/lib/kubelet/pods/80e45018-1682-47cd-9523-dbeaf35a39cd/volumes/kubernetes.io~projected/kube-api-access-vdblb major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9092fa02-cc22-41a2-b4ee-070298ecab99/volumes/kubernetes.io~projected/kube-api-access-4p6bb:{mountpoint:/var/lib/kubelet/pods/9092fa02-cc22-41a2-b4ee-070298ecab99/volumes/kubernetes.io~projected/kube-api-access-4p6bb major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90df09a3-8706-4883-b711-be63e8e9f191/volumes/kubernetes.io~projected/kube-api-access-hhnjp:{mountpoint:/var/lib/kubelet/pods/90df09a3-8706-4883-b711-be63e8e9f191/volumes/kubernetes.io~projected/kube-api-access-hhnjp major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92981ab7-a41b-482f-98c3-a328e5e8d6f0/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/92981ab7-a41b-482f-98c3-a328e5e8d6f0/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92981ab7-a41b-482f-98c3-a328e5e8d6f0/volumes/kubernetes.io~projected/kube-api-access-dth6b:{mountpoint:/var/lib/kubelet/pods/92981ab7-a41b-482f-98c3-a328e5e8d6f0/volumes/kubernetes.io~projected/kube-api-access-dth6b major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92981ab7-a41b-482f-98c3-a328e5e8d6f0/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/92981ab7-a41b-482f-98c3-a328e5e8d6f0/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92981ab7-a41b-482f-98c3-a328e5e8d6f0/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/92981ab7-a41b-482f-98c3-a328e5e8d6f0/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92f38b64-26fb-42d6-826a-94cd37e8b668/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/92f38b64-26fb-42d6-826a-94cd37e8b668/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d148c95-78ae-43af-8942-460853970292/volumes/kubernetes.io~projected/kube-api-access-cr2vr:{mountpoint:/var/lib/kubelet/pods/9d148c95-78ae-43af-8942-460853970292/volumes/kubernetes.io~projected/kube-api-access-cr2vr major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a11d470f-d198-42ec-bee1-a900c4814473/volumes/kubernetes.io~projected/kube-api-access-v9z9l:{mountpoint:/var/lib/kubelet/pods/a11d470f-d198-42ec-bee1-a900c4814473/volumes/kubernetes.io~projected/kube-api-access-v9z9l major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a20a00eb-cde2-4df4-a191-0502fcaf288d/volumes/kubernetes.io~projected/kube-api-access-5dh8q:{mountpoint:/var/lib/kubelet/pods/a20a00eb-cde2-4df4-a191-0502fcaf288d/volumes/kubernetes.io~projected/kube-api-access-5dh8q major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9/volumes/kubernetes.io~projected/kube-api-access-l74z8:{mountpoint:/var/lib/kubelet/pods/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9/volumes/kubernetes.io~projected/kube-api-access-l74z8 major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9/volumes/kubernetes.io~secret/serving-cert major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b94a90ae-e06a-4e60-a48e-b567ce6e4edd/volumes/kubernetes.io~projected/kube-api-access-dmg9f:{mountpoint:/var/lib/kubelet/pods/b94a90ae-e06a-4e60-a48e-b567ce6e4edd/volumes/kubernetes.io~projected/kube-api-access-dmg9f major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9849d01-613f-439d-86de-b517ee4a3341/volumes/kubernetes.io~projected/kube-api-access-jvl8c:{mountpoint:/var/lib/kubelet/pods/b9849d01-613f-439d-86de-b517ee4a3341/volumes/kubernetes.io~projected/kube-api-access-jvl8c major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be409e3c-3420-4250-bfa7-3b01ecd69d9b/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/be409e3c-3420-4250-bfa7-3b01ecd69d9b/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be409e3c-3420-4250-bfa7-3b01ecd69d9b/volumes/kubernetes.io~projected/kube-api-access-knktl:{mountpoint:/var/lib/kubelet/pods/be409e3c-3420-4250-bfa7-3b01ecd69d9b/volumes/kubernetes.io~projected/kube-api-access-knktl major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be409e3c-3420-4250-bfa7-3b01ecd69d9b/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/be409e3c-3420-4250-bfa7-3b01ecd69d9b/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be409e3c-3420-4250-bfa7-3b01ecd69d9b/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/be409e3c-3420-4250-bfa7-3b01ecd69d9b/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8842256-47fc-4dc5-91f6-65938f410080/volumes/kubernetes.io~projected/kube-api-access-5hp67:{mountpoint:/var/lib/kubelet/pods/d8842256-47fc-4dc5-91f6-65938f410080/volumes/kubernetes.io~projected/kube-api-access-5hp67 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1c10266-f60d-4b43-b2ee-cef69669bc7b/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/e1c10266-f60d-4b43-b2ee-cef69669bc7b/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1c10266-f60d-4b43-b2ee-cef69669bc7b/volumes/kubernetes.io~projected/kube-api-access-4tsxg:{mountpoint:/var/lib/kubelet/pods/e1c10266-f60d-4b43-b2ee-cef69669bc7b/volumes/kubernetes.io~projected/kube-api-access-4tsxg major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1c10266-f60d-4b43-b2ee-cef69669bc7b/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/e1c10266-f60d-4b43-b2ee-cef69669bc7b/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e438abba-e5b6-41bb-bc67-1a1d322e6534/volumes/kubernetes.io~projected/kube-api-access-6hq7p:{mountpoint:/var/lib/kubelet/pods/e438abba-e5b6-41bb-bc67-1a1d322e6534/volumes/kubernetes.io~projected/kube-api-access-6hq7p major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e954d56b-ace6-4539-a467-486c1e889886/volumes/kubernetes.io~projected/kube-api-access-qznzq:{mountpoint:/var/lib/kubelet/pods/e954d56b-ace6-4539-a467-486c1e889886/volumes/kubernetes.io~projected/kube-api-access-qznzq major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e954d56b-ace6-4539-a467-486c1e889886/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e954d56b-ace6-4539-a467-486c1e889886/volumes/kubernetes.io~secret/serving-cert major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f416f20b-a466-4f3d-bd2e-bb825b189067/volumes/kubernetes.io~projected/kube-api-access-m5c5k:{mountpoint:/var/lib/kubelet/pods/f416f20b-a466-4f3d-bd2e-bb825b189067/volumes/kubernetes.io~projected/kube-api-access-m5c5k major:0 minor:64 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c27f506dfebd15bd5560e0b027fa5e229a995fbcf2021482c80a91ac75199533/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/bc467856764310c706001edf2167aa07151ffd426cf1f9c98e1b9f44524f1b46/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/ad3e2cebc5fb83e860853705df9759a1d1a8a399c8e6afcb1d870650d24b946f/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/829c7a9ac472b57707c6f56db656cefd223202fa31b93645f6380f23c259514f/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/439b827ad46ee85893e2ccd17e4b649099c538b53d27117c37222c8bf4e6158a/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/404ee327546e29f56122eeb830459131de7b0e7266db8c5771a1a421457d3dfa/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/0496a69d8cbc1dea0a21ce8403cb11cc108db8c5da758cdd8b23c5ab61d9d772/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/4c2c57d522e2162494875734048d1f25854e2e900ef9a7b37d0bd6ce57678632/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/f95ee759f5c355886303a720acb9c0f250ec5d260d982fdc28b7a3f8099cd2ea/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-135:{mountpoint:/var/lib/containers/storage/overlay/850f963ff51720025c8fb5144dd41f7ebc0c1558306fdb07c23c3f813848e82d/merged major:0 minor:135 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/dac5ea8692def5f8b6848e89d948e74b63c05c1b020002214970d4003886a484/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/3d773b5612241cea959afa310e50aebdc4c87844b8c13e41aea6f4edff67a86a/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/32be1182b5a3adaece6eec9bc9c993f8ebcd2ba87e1e6b7f9b15111f49c78cea/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/1d1af49cc970278e6e93b6c41fd433986e323477c9bbc64d8926623dd3fb87e8/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/7a9fd5b2c668b59d6dcfecf71929ba22fc3244579c7c84f52db9414579303b68/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/e91b30e3af3aa8ed5952aca3b9b55d89def9e048bb9a7ff1b88cb0b4b4b2ac58/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/262e756c3deb9c22aaedd82ae4360019fee5274d7dcb2e8611b4d99e90d829e8/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/72110fdeecde058398d0155cbc296689c6eaf67c13af31bb7660565353fef6d0/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/72c263b63fa2a81ea8d0ed3b759b712d347b8815b7cac4ca824e57fecc877277/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/5a54d9f7bd3d4e55c9aa563cfc22524a918eb04ca6edf0926303de27267fbca5/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/f79cd4ac877cf2abc5103334d266fd5ea582128d30d4f4fa2ffa8ff7901aef70/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/13325ba52e478cf82a368f29bd522672a188c87d58e7134a0a636498ead29038/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/c1ab89c42991a896d0615c7f88c1610e12f6b34e92502b2e3eddf8888879410c/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/0064ef1a410b6d584e3acc72f108c0011c73a5353da67d93715a78171f157719/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/3e5dbc8dc45bc2192f1663165c2b65fe33031b5cb28a42603b4853164a1a5e80/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/093ffd0291064f81c80ee738d8eacdf5cf9bb20e80b289532c89fc570b7bcf78/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/49b75c131911840ca5c6b0e3b3cd027a74c81774e72b357bfa014d25c6b73361/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/537156351a4cfc5719b4c06840c86c551cdbbb0b955c7b08ab0c1dd49c730caf/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/60b0b4870ea77f9e90c4807471560ef6101ddeb01537d000fd554e441c2d42df/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/d377da2fd790293352f43838a8f3e0fed8d896dfa392f8199c12bd776d9ab0bd/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/7770094599e2c34281f097bacffc394ae02d5a19b0fd65125025560482659859/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/871979b15605df2771155a8c1e31ec5817c8f7f0b590087f3f4362ac246bdd82/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/9dc05332b13a4bebed4380a7810d9f3d99102b3e6d77fea6d9d13dc0c1d6ef17/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/55bb158cf1b3d6fa02e3c3b18d41d15c611d60a6abfe6980ada7c41f085c9f95/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/f0435ce99017dbc53adba5f532efd12db8b27dce4447d9aa442ed83c8c82b95b/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/f687f0e47215e72e17ca70793d5e8f92af018a7112ac8d4ef72a2436795bf18a/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/f1e0cc25f8d0951a53cb694d524cf3ff939f6ef62e4301d389ebbd2886ed6967/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/f596fb73e742d1394c86366633f91460615b41a49645bd8f8a67e1dbb758a14d/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/2d6e9c422ba8de6c63190ecc4d1c628695ff6d2a3ea35e40663c47eb7614e275/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/61a4b555a75211295baf8bbdfdfede0c19089f8906fd033302c8a8901754c2aa/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/94da2635c0bd83be2614f0c4ce90b9fdded7525de0788f08a4b67720e76f26c9/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/e97065fcdcb66b0b0709d494f1e69c5d2b2855d22cebe8aee0c823a4b86e6058/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/864a1b1df9ad9b3fd5b0583be5d6320b5b188e21d57f0ef18a12630767766431/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/0ae56a7c09f2da07a7c4f8ef26b5970f70f34db7fe06bcf5fab48a2d872d9c32/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/871140812ceb30c7041196c9e2837f0eed3cc3ef0e456e74c41d5aa2262f2433/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/be8344c890ca42c1c80491f28562971e2d1fc69395d382668df061ee59c9a587/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/fc1efb80d2b9a541e985dfcdc046d1df99117ea54bbae2d4d55b7e70e8531646/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/869e23ab688fada2aba9a41e2b53d54391b7403b06a5dbf69b7fe9a70b0e73fa/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/52a91cb1b671a90c10832f35c7923d51318f4c2039bab19f289cac9395d7f492/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/56f40ce1d79e64eb218eb87f5cb10b0f5863b39e06e6fce4393718d20bdd4f4c/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/37a6c41c446ab539fdecaba5d61c3b984210d6ae650df673c1f8c1fdbdda10e1/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/f407829e157f81dd4651534b725f54430ebfb66eb85c0f227a24f883731090ee/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/bc59c1931c8b24d99803f57a4fb140bc2281dd246be82e82c4d069abf69718bc/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/1b6d3d73832dd551bd38dba877d4a2909898368adbaccc07dbd9cfece3a35ee8/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/a1aae1121e61c4fff3e0ebe59ea719e8947a3e5a9b5715e5d1cf8af8ac57a781/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/d930653b5dd7711a6f97b46e68118aace4f61fe5eabf2a6e27cdc8e582f33c5b/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 18:36:38.782368 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.781980    4924 manager.go:217] Machine: {Timestamp:2026-10-30 18:36:38.78152719 +0000 UTC m=+0.058577626 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097163 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec270e2ddf6e3bb5b0636a821049e376 SystemUUID:ec270e2d-df6e-3bb5-b063-6a821049e376 BootID:1aa03d8d-b7ae-44f7-a873-6fb1856fd21e Filesystems:[{Device:/run/containers/storage/overlay-containers/f863f9d519de3c1bdafab1e8717dd8731b7fafdd7c9c7b23a372d5d64b18c521/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e438abba-e5b6-41bb-bc67-1a1d322e6534/volumes/kubernetes.io~projected/kube-api-access-6hq7p DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e099f4ea8091b1a7d81d6f31319a778089bcb24400d3e279d44461982cda02d/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/05fe2f04-29f7-4999-87ef-b30035b34883/volumes/kubernetes.io~projected/kube-api-access-mmjgg DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/be409e3c-3420-4250-bfa7-3b01ecd69d9b/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e1c10266-f60d-4b43-b2ee-cef69669bc7b/volumes/kubernetes.io~projected/kube-api-access-4tsxg DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b9849d01-613f-439d-86de-b517ee4a3341/volumes/kubernetes.io~projected/kube-api-access-jvl8c DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/47de5c3c76d6b584d3f74b82db93dff272d550428be9da41ddfe3619fa3ec2d5/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/14de059988139725a952bc59f04bd0c84c5561ad20b6de9d807f2577d2cb131a/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65beb5a5-1df3-42c8-9e14-769b4ad47255/volumes/kubernetes.io~projected/kube-api-access-jpqmj DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/05fe2f04-29f7-4999-87ef-b30035b34883/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74a85732-6555-48bf-9ece-4e863cd1fb93/volumes/kubernetes.io~projected/kube-api-access-qkl9j DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/92981ab7-a41b-482f-98c3-a328e5e8d6f0/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/60e0e397-bb2c-4637-9376-c5af871666cb/volumes/kubernetes.io~projected/kube-api-access-v526v DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92981ab7-a41b-482f-98c3-a328e5e8d6f0/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/be409e3c-3420-4250-bfa7-3b01ecd69d9b/volumes/kubernetes.io~projected/kube-api-access-knktl DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9/volumes/kubernetes.io~projected/kube-api-access-l74z8 DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9746669de1d07f5c2c427a3f47b253acf5cab3b53fd2d2afe6019626e24c8ba/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/60e0e397-bb2c-4637-9376-c5af871666cb/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/21b6f672-2dc5-4523-9dfa-6dfaa30a8bac/volumes/kubernetes.io~projected/kube-api-access-ks7sd DeviceMajor:0 DeviceMinor:232 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e0da5dc-3a50-414f-99aa-0a8bbe303a31/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e954d56b-ace6-4539-a467-486c1e889886/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ecb952775b583a3883cc36ec2b142c98891e7d39ff126a4edcd4a1eef4131f3/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b7dd88b77a31d2d61979b2e9b046603cc78f3aae97bc95a0bac4ddbf3309384/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e1c10266-f60d-4b43-b2ee-cef69669bc7b/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6df0f1e7-07b7-4231-bc70-ba81aef29856/volumes/kubernetes.io~projected/kube-api-access-4fdth DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f46bb25af1cc8f5165ee09c930e7a01203af203a8515498da17f95919c2cc4f4/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3aaf547da0a47eca32ce390a3406034d7022e20fea73e56992f72657a781c2dd/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-135 DeviceMajor:0 DeviceMinor:135 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12928ad7-4038-473a-a4aa-be1a6a42eabc/volumes/kubernetes.io~projected/kube-api-access-qcz45 DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a5a5bcab19b8f9cd4474e125f5e0e25e6671882e7c8e722fbeb629c8f8f3821/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278/volumes/kubernetes.io~projected/kube-api-access-vb856 DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90724e59845a496535c3fa041fad3cb95848b518bd1225c142dd012354e9e99e/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/92981ab7-a41b-482f-98c3-a328e5e8d6f0/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f416f20b-a466-4f3d-bd2e-bb825b189067/volumes/kubernetes.io~projected/kube-api-access-m5c5k DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80977e6181a3c742059b0adae7b88dbd4190d264ac3da7c81df5c9a38370a97d/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/3935b0fd-69be-4272-adc5-ecfbc7af04ae/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9092fa02-cc22-41a2-b4ee-070298ecab99/volumes/kubernetes.io~projected/kube-api-access-4p6bb DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aaffbe47de8f1818df45a736effd1ad88bf5a61518d31a7a73b2695a9f07e400/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d42e55117f9bd1279beabb43c4a75ceb656285a170dffb6bdc9e7531b4666440/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf414281ada08cae7668511c127437eedad35ede1b4ec220a53de48da3f35ebe/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92981ab7-a41b-482f-98c3-a328e5e8d6f0/volumes/kubernetes.io~projected/kube-api-access-dth6b DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e4755db47b4602d74bc92410b4139a0c03ec31fe285b49cc74c99928d8351c06/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e954d56b-ace6-4539-a467-486c1e889886/volumes/kubernetes.io~projected/kube-api-access-qznzq DeviceMajor:0 DeviceMinor:234 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b982014ee3a54d7d3cb4c6619b1f3d0b15954c25fd5bac0f95a58c78a3541556/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ef02004-71fc-4464-9550-bd094d27f21f/volumes/kubernetes.io~projected/kube-api-access-dx8hm DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9d148c95-78ae-43af-8942-460853970292/volumes/kubernetes.io~projected/kube-api-access-cr2vr DeviceMajor:0 DeviceMinor:314 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/80e45018-1682-47cd-9523-dbeaf35a39cd/volumes/kubernetes.io~projected/kube-api-access-vdblb DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e1c10266-f60d-4b43-b2ee-cef69669bc7b/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/a20a00eb-cde2-4df4-a191-0502fcaf288d/volumes/kubernetes.io~projected/kube-api-access-5dh8q DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/05fe2f04-29f7-4999-87ef-b30035b34883/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92f38b64-26fb-42d6-826a-94cd37e8b668/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0648f559-00ac-4de0-8b9f-ce02d1b657a9/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5edaeb3ad7c0c9d0fe1f145eeace4b609b052a02d7b21055e42ca516feff8f11/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/022157ce1cf92e0afd69fe9ec69a54f1aa1da7587493d5ce91ef80846f269ccf/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a11d470f-d198-42ec-bee1-a900c4814473/volumes/kubernetes.io~projected/kube-api-access-v9z9l DeviceMajor:0 DeviceMinor:339 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d8842256-47fc-4dc5-91f6-65938f410080/volumes/kubernetes.io~projected/kube-api-access-5hp67 DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a0551ed10d5edfadfa22b5ba6600154ed5ee9ff3090c925802c7867c7e97bfec/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21b6f672-2dc5-4523-9dfa-6dfaa30a8bac/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b94a90ae-e06a-4e60-a48e-b567ce6e4edd/volumes/kubernetes.io~projected/kube-api-access-dmg9f DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5e736af7cb22e7e295d2c82ed4c7548f4498a44bbd419920772c5a90e05ce4e/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0648f559-00ac-4de0-8b9f-ce02d1b657a9/volumes/kubernetes.io~projected/kube-api-access-k8ggz DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c56149a6c66eaa82146cbbab520741d290fbb1283425fd10aa2409a3f28accf/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3935b0fd-69be-4272-adc5-ecfbc7af04ae/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3935b0fd-69be-4272-adc5-ecfbc7af04ae/volumes/kubernetes.io~projected/kube-api-access-g6bwn DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/be409e3c-3420-4250-bfa7-3b01ecd69d9b/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/be409e3c-3420-4250-bfa7-3b01ecd69d9b/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5d05b776-9750-4c5b-8d91-4cf27329a1fc/volumes/kubernetes.io~projected/kube-api-access-2xg8f DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f8e7bbb9d66216640a2e4edc5ffdb3166448ec13bc4800d1523680d155f3634b/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/90df09a3-8706-4883-b711-be63e8e9f191/volumes/kubernetes.io~projected/kube-api-access-hhnjp DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:022157ce1cf92e0 MacAddress:9e:92:e6:01:67:cf Speed:10000 Mtu:8901} {Name:14de05998813972 MacAddress:26:2d:dc:e3:ae:07 Speed:10000 Mtu:8901} {Name:2e099f4ea8091b1 MacAddress:0e:a7:84:e2:f2:57 Speed:10000 Mtu:8901} {Name:3aaf547da0a47ec MacAddress:3e:36:a0:89:69:f2 Speed:10000 Mtu:8901} {Name:5edaeb3ad7c0c9d MacAddress:12:b6:14:c5:49:33 Speed:10000 Mtu:8901} {Name:6b7dd88b77a31d2 MacAddress:4e:04:83:2e:e4:fb Speed:10000 Mtu:8901} {Name:8ecb952775b583a MacAddress:ba:3a:ac:14:89:4d Speed:10000 Mtu:8901} {Name:a0551ed10d5edfa MacAddress:3a:a1:11:ca:d1:3b Speed:10000 Mtu:8901} {Name:aaffbe47de8f181 MacAddress:7a:c1:69:ee:df:32 Speed:10000 Mtu:8901} {Name:b982014ee3a54d7 MacAddress:8a:33:68:3f:1c:31 Speed:10000 Mtu:8901} {Name:bf414281ada08ca MacAddress:e2:1a:9d:b5:91:c9 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:ff:55:2e:ab Speed:0 Mtu:9001} {Name:br-int MacAddress:b6:00:c6:f2:5c:a2 Speed:0 Mtu:8901} {Name:ens5 MacAddress:06:ff:ff:55:2e:ab Speed:0 Mtu:9001} {Name:f46bb25af1cc8f5 MacAddress:b2:5f:1b:42:16:2b Speed:10000 Mtu:8901} {Name:f8e7bbb9d662166 MacAddress:fa:62:6f:48:8c:f5 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:36:a8:03:ef:a4:f2 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:be:fb:b9:54:73:c2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 18:36:38.782368 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.782317    4924 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 18:36:38.782368 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.782358    4924 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 18:36:38.782652 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.782632    4924 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 18:36:38.782783 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.782653    4924 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-224.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 18:36:38.782826 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.782785    4924 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 18:36:38.782826 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.782794    4924 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 18:36:38.782826 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.782804    4924 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:36:38.782826 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.782814    4924 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:36:38.782928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.782893    4924 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 18:36:38.783058 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.783048    4924 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 18:36:38.783180 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.783171    4924 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 18:36:38.783216 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.783208    4924 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 18:36:38.783244 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.783225    4924 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 18:36:38.783244 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.783235    4924 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 18:36:38.783244 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.783244    4924 kubelet.go:405] "Adding apiserver pod source"
Oct 30 18:36:38.783324 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.783252    4924 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 18:36:38.783972 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.783957    4924 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 18:36:38.783972 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.783970    4924 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:36:38.784483 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.784456    4924 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 18:36:38.788364 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.788346    4924 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 18:36:38.788364 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.788361    4924 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:36:38.788486 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.788452    4924 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 18:36:38.788637 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.788626    4924 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 18:36:38.788680 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.788655    4924 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 18:36:38.788746 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.788737    4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 18:36:38.788776 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.788751    4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 18:36:38.788776 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.788758    4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 18:36:38.788776 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.788763    4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 18:36:38.788776 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.788769    4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 18:36:38.788776 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.788775    4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 18:36:38.788911 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.788781    4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 18:36:38.788911 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.788786    4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 18:36:38.788911 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.788793    4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 18:36:38.788911 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.788799    4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 18:36:38.788911 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.788811    4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 18:36:38.788911 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.788821    4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 18:36:38.788911 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.788843    4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 18:36:38.788911 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.788849    4924 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 18:36:38.792222 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.792208    4924 server.go:1263] "Started kubelet"
Oct 30 18:36:38.792305 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.792280    4924 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 18:36:38.792356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.792310    4924 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 18:36:38.792403 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.792379    4924 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 18:36:38.792774 ip-10-0-131-224 systemd[1]: Started Kubernetes Kubelet.
Oct 30 18:36:38.792986 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.792805    4924 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 18:36:38.794299 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.794277    4924 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 18:36:38.797268 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.795752    4924 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 18:36:38.797268 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.795773    4924 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 18:36:38.797268 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.795820    4924 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:30:59 +0000 UTC" deadline="2028-07-26 20:48:48.902894228 +0000 UTC"
Oct 30 18:36:38.797268 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.797118    4924 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15242h12m10.105783986s"
Oct 30 18:36:38.797268 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.795896    4924 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 18:36:38.797268 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:38.795888    4924 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-224.us-west-2.compute.internal\" not found"
Oct 30 18:36:38.797268 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.795883    4924 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 18:36:38.797268 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.797160    4924 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 18:36:38.799039 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.799021    4924 factory.go:55] Registering systemd factory
Oct 30 18:36:38.799039 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.799041    4924 factory.go:223] Registration of the systemd container factory successfully
Oct 30 18:36:38.800919 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.799508    4924 factory.go:153] Registering CRI-O factory
Oct 30 18:36:38.800919 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.799528    4924 factory.go:223] Registration of the crio container factory successfully
Oct 30 18:36:38.800919 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.799581    4924 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 18:36:38.800919 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.799612    4924 factory.go:103] Registering Raw factory
Oct 30 18:36:38.800919 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.799633    4924 manager.go:1199] Started watching for new ooms in manager
Oct 30 18:36:38.800919 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.800042    4924 manager.go:319] Starting recovery of all containers
Oct 30 18:36:38.806276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.805817    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12928ad7-4038-473a-a4aa-be1a6a42eabc" volumeName="kubernetes.io/empty-dir/12928ad7-4038-473a-a4aa-be1a6a42eabc-tmp-dir" seLinuxMountContext=""
Oct 30 18:36:38.806276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.805867    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a11d470f-d198-42ec-bee1-a900c4814473" volumeName="kubernetes.io/projected/a11d470f-d198-42ec-bee1-a900c4814473-kube-api-access-v9z9l" seLinuxMountContext=""
Oct 30 18:36:38.806276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.805884    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9" volumeName="kubernetes.io/configmap/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 18:36:38.806276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.805904    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be409e3c-3420-4250-bfa7-3b01ecd69d9b" volumeName="kubernetes.io/secret/be409e3c-3420-4250-bfa7-3b01ecd69d9b-installation-pull-secrets" seLinuxMountContext=""
Oct 30 18:36:38.806276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.805917    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e0da5dc-3a50-414f-99aa-0a8bbe303a31" volumeName="kubernetes.io/configmap/1e0da5dc-3a50-414f-99aa-0a8bbe303a31-konnectivity-ca" seLinuxMountContext=""
Oct 30 18:36:38.806276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.805934    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9092fa02-cc22-41a2-b4ee-070298ecab99" volumeName="kubernetes.io/projected/9092fa02-cc22-41a2-b4ee-070298ecab99-kube-api-access-4p6bb" seLinuxMountContext=""
Oct 30 18:36:38.806276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.805946    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be409e3c-3420-4250-bfa7-3b01ecd69d9b" volumeName="kubernetes.io/empty-dir/be409e3c-3420-4250-bfa7-3b01ecd69d9b-ca-trust-extracted" seLinuxMountContext=""
Oct 30 18:36:38.806276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.805970    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e954d56b-ace6-4539-a467-486c1e889886" volumeName="kubernetes.io/secret/e954d56b-ace6-4539-a467-486c1e889886-serving-cert" seLinuxMountContext=""
Oct 30 18:36:38.806276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.805991    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0648f559-00ac-4de0-8b9f-ce02d1b657a9" volumeName="kubernetes.io/projected/0648f559-00ac-4de0-8b9f-ce02d1b657a9-kube-api-access-k8ggz" seLinuxMountContext=""
Oct 30 18:36:38.806276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806004    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80e45018-1682-47cd-9523-dbeaf35a39cd" volumeName="kubernetes.io/projected/80e45018-1682-47cd-9523-dbeaf35a39cd-kube-api-access-vdblb" seLinuxMountContext=""
Oct 30 18:36:38.806276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806021    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9849d01-613f-439d-86de-b517ee4a3341" volumeName="kubernetes.io/projected/b9849d01-613f-439d-86de-b517ee4a3341-kube-api-access-jvl8c" seLinuxMountContext=""
Oct 30 18:36:38.806276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806034    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65beb5a5-1df3-42c8-9e14-769b4ad47255" volumeName="kubernetes.io/configmap/65beb5a5-1df3-42c8-9e14-769b4ad47255-cni-binary-copy" seLinuxMountContext=""
Oct 30 18:36:38.806276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806053    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12928ad7-4038-473a-a4aa-be1a6a42eabc" volumeName="kubernetes.io/configmap/12928ad7-4038-473a-a4aa-be1a6a42eabc-config-volume" seLinuxMountContext=""
Oct 30 18:36:38.806276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806082    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ef02004-71fc-4464-9550-bd094d27f21f" volumeName="kubernetes.io/projected/5ef02004-71fc-4464-9550-bd094d27f21f-kube-api-access-dx8hm" seLinuxMountContext=""
Oct 30 18:36:38.806276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806106    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60e0e397-bb2c-4637-9376-c5af871666cb" volumeName="kubernetes.io/configmap/60e0e397-bb2c-4637-9376-c5af871666cb-config" seLinuxMountContext=""
Oct 30 18:36:38.806276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806136    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60e0e397-bb2c-4637-9376-c5af871666cb" volumeName="kubernetes.io/projected/60e0e397-bb2c-4637-9376-c5af871666cb-kube-api-access-v526v" seLinuxMountContext=""
Oct 30 18:36:38.806276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806150    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be409e3c-3420-4250-bfa7-3b01ecd69d9b" volumeName="kubernetes.io/configmap/be409e3c-3420-4250-bfa7-3b01ecd69d9b-trusted-ca" seLinuxMountContext=""
Oct 30 18:36:38.806276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806223    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8842256-47fc-4dc5-91f6-65938f410080" volumeName="kubernetes.io/projected/d8842256-47fc-4dc5-91f6-65938f410080-kube-api-access-5hp67" seLinuxMountContext=""
Oct 30 18:36:38.806276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806253    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0648f559-00ac-4de0-8b9f-ce02d1b657a9" volumeName="kubernetes.io/secret/0648f559-00ac-4de0-8b9f-ce02d1b657a9-serving-cert" seLinuxMountContext=""
Oct 30 18:36:38.806276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806263    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3935b0fd-69be-4272-adc5-ecfbc7af04ae" volumeName="kubernetes.io/projected/3935b0fd-69be-4272-adc5-ecfbc7af04ae-kube-api-access-g6bwn" seLinuxMountContext=""
Oct 30 18:36:38.806276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806278    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65beb5a5-1df3-42c8-9e14-769b4ad47255" volumeName="kubernetes.io/configmap/65beb5a5-1df3-42c8-9e14-769b4ad47255-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 18:36:38.806276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806286    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92981ab7-a41b-482f-98c3-a328e5e8d6f0" volumeName="kubernetes.io/configmap/92981ab7-a41b-482f-98c3-a328e5e8d6f0-trusted-ca" seLinuxMountContext=""
Oct 30 18:36:38.806276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806297    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9" volumeName="kubernetes.io/configmap/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-service-ca-bundle" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806305    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9" volumeName="kubernetes.io/empty-dir/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-tmp" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806313    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be409e3c-3420-4250-bfa7-3b01ecd69d9b" volumeName="kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-bound-sa-token" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806341    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e954d56b-ace6-4539-a467-486c1e889886" volumeName="kubernetes.io/configmap/e954d56b-ace6-4539-a467-486c1e889886-config" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806351    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74a85732-6555-48bf-9ece-4e863cd1fb93" volumeName="kubernetes.io/configmap/74a85732-6555-48bf-9ece-4e863cd1fb93-cni-binary-copy" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806367    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92981ab7-a41b-482f-98c3-a328e5e8d6f0" volumeName="kubernetes.io/secret/92981ab7-a41b-482f-98c3-a328e5e8d6f0-installation-pull-secrets" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806384    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a11d470f-d198-42ec-bee1-a900c4814473" volumeName="kubernetes.io/empty-dir/a11d470f-d198-42ec-bee1-a900c4814473-data-volume" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806396    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f416f20b-a466-4f3d-bd2e-bb825b189067" volumeName="kubernetes.io/configmap/f416f20b-a466-4f3d-bd2e-bb825b189067-iptables-alerter-script" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806407    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90df09a3-8706-4883-b711-be63e8e9f191" volumeName="kubernetes.io/empty-dir/90df09a3-8706-4883-b711-be63e8e9f191-tmp-dir" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806436    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1c10266-f60d-4b43-b2ee-cef69669bc7b" volumeName="kubernetes.io/configmap/e1c10266-f60d-4b43-b2ee-cef69669bc7b-ovnkube-config" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806741    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21b6f672-2dc5-4523-9dfa-6dfaa30a8bac" volumeName="kubernetes.io/secret/21b6f672-2dc5-4523-9dfa-6dfaa30a8bac-cert" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806765    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3935b0fd-69be-4272-adc5-ecfbc7af04ae" volumeName="kubernetes.io/empty-dir/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-tuned" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806778    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74a85732-6555-48bf-9ece-4e863cd1fb93" volumeName="kubernetes.io/projected/74a85732-6555-48bf-9ece-4e863cd1fb93-kube-api-access-qkl9j" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806791    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9" volumeName="kubernetes.io/projected/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-kube-api-access-l74z8" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806804    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90df09a3-8706-4883-b711-be63e8e9f191" volumeName="kubernetes.io/projected/90df09a3-8706-4883-b711-be63e8e9f191-kube-api-access-hhnjp" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806815    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92981ab7-a41b-482f-98c3-a328e5e8d6f0" volumeName="kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-kube-api-access-dth6b" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806827    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92981ab7-a41b-482f-98c3-a328e5e8d6f0" volumeName="kubernetes.io/secret/92981ab7-a41b-482f-98c3-a328e5e8d6f0-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806840    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9" volumeName="kubernetes.io/empty-dir/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-snapshots" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806853    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be409e3c-3420-4250-bfa7-3b01ecd69d9b" volumeName="kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-kube-api-access-knktl" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806866    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be409e3c-3420-4250-bfa7-3b01ecd69d9b" volumeName="kubernetes.io/secret/be409e3c-3420-4250-bfa7-3b01ecd69d9b-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806890    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1c10266-f60d-4b43-b2ee-cef69669bc7b" volumeName="kubernetes.io/projected/e1c10266-f60d-4b43-b2ee-cef69669bc7b-kube-api-access-4tsxg" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806901    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e954d56b-ace6-4539-a467-486c1e889886" volumeName="kubernetes.io/projected/e954d56b-ace6-4539-a467-486c1e889886-kube-api-access-qznzq" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806912    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05fe2f04-29f7-4999-87ef-b30035b34883" volumeName="kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-default-certificate" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806923    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0648f559-00ac-4de0-8b9f-ce02d1b657a9" volumeName="kubernetes.io/configmap/0648f559-00ac-4de0-8b9f-ce02d1b657a9-config" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806940    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b0aa783-d4ab-490c-a5f7-d7da5ad8d278" volumeName="kubernetes.io/projected/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278-kube-api-access-vb856" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806952    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e0da5dc-3a50-414f-99aa-0a8bbe303a31" volumeName="kubernetes.io/secret/1e0da5dc-3a50-414f-99aa-0a8bbe303a31-agent-certs" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806962    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d05b776-9750-4c5b-8d91-4cf27329a1fc" volumeName="kubernetes.io/projected/5d05b776-9750-4c5b-8d91-4cf27329a1fc-kube-api-access-2xg8f" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806973    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a20a00eb-cde2-4df4-a191-0502fcaf288d" volumeName="kubernetes.io/projected/a20a00eb-cde2-4df4-a191-0502fcaf288d-kube-api-access-5dh8q" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806984    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b94a90ae-e06a-4e60-a48e-b567ce6e4edd" volumeName="kubernetes.io/projected/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-kube-api-access-dmg9f" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.806996    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1c10266-f60d-4b43-b2ee-cef69669bc7b" volumeName="kubernetes.io/configmap/e1c10266-f60d-4b43-b2ee-cef69669bc7b-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807011    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92981ab7-a41b-482f-98c3-a328e5e8d6f0" volumeName="kubernetes.io/configmap/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-certificates" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807024    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b63e008-3071-4593-bba8-3bcc673b5dc2" volumeName="kubernetes.io/configmap/9b63e008-3071-4593-bba8-3bcc673b5dc2-nginx-conf" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807038    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65beb5a5-1df3-42c8-9e14-769b4ad47255" volumeName="kubernetes.io/projected/65beb5a5-1df3-42c8-9e14-769b4ad47255-kube-api-access-jpqmj" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807046    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12928ad7-4038-473a-a4aa-be1a6a42eabc" volumeName="kubernetes.io/projected/12928ad7-4038-473a-a4aa-be1a6a42eabc-kube-api-access-qcz45" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807059    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65beb5a5-1df3-42c8-9e14-769b4ad47255" volumeName="kubernetes.io/configmap/65beb5a5-1df3-42c8-9e14-769b4ad47255-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807085    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be409e3c-3420-4250-bfa7-3b01ecd69d9b" volumeName="kubernetes.io/configmap/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-certificates" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807097    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f416f20b-a466-4f3d-bd2e-bb825b189067" volumeName="kubernetes.io/projected/f416f20b-a466-4f3d-bd2e-bb825b189067-kube-api-access-m5c5k" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807108    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05fe2f04-29f7-4999-87ef-b30035b34883" volumeName="kubernetes.io/projected/05fe2f04-29f7-4999-87ef-b30035b34883-kube-api-access-mmjgg" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807129    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21b6f672-2dc5-4523-9dfa-6dfaa30a8bac" volumeName="kubernetes.io/projected/21b6f672-2dc5-4523-9dfa-6dfaa30a8bac-kube-api-access-ks7sd" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807141    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3935b0fd-69be-4272-adc5-ecfbc7af04ae" volumeName="kubernetes.io/empty-dir/3935b0fd-69be-4272-adc5-ecfbc7af04ae-tmp" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807161    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60e0e397-bb2c-4637-9376-c5af871666cb" volumeName="kubernetes.io/secret/60e0e397-bb2c-4637-9376-c5af871666cb-serving-cert" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807172    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6df0f1e7-07b7-4231-bc70-ba81aef29856" volumeName="kubernetes.io/projected/6df0f1e7-07b7-4231-bc70-ba81aef29856-kube-api-access-4fdth" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807184    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74a85732-6555-48bf-9ece-4e863cd1fb93" volumeName="kubernetes.io/configmap/74a85732-6555-48bf-9ece-4e863cd1fb93-multus-daemon-config" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807194    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92f38b64-26fb-42d6-826a-94cd37e8b668" volumeName="kubernetes.io/secret/92f38b64-26fb-42d6-826a-94cd37e8b668-original-pull-secret" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807214    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9" volumeName="kubernetes.io/secret/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-serving-cert" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807225    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60e0e397-bb2c-4637-9376-c5af871666cb" volumeName="kubernetes.io/configmap/60e0e397-bb2c-4637-9376-c5af871666cb-trusted-ca" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807237    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80e45018-1682-47cd-9523-dbeaf35a39cd" volumeName="kubernetes.io/configmap/80e45018-1682-47cd-9523-dbeaf35a39cd-telemetry-config" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807247    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d148c95-78ae-43af-8942-460853970292" volumeName="kubernetes.io/projected/9d148c95-78ae-43af-8942-460853970292-kube-api-access-cr2vr" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807260    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1c10266-f60d-4b43-b2ee-cef69669bc7b" volumeName="kubernetes.io/secret/e1c10266-f60d-4b43-b2ee-cef69669bc7b-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807287    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05fe2f04-29f7-4999-87ef-b30035b34883" volumeName="kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-stats-auth" seLinuxMountContext=""
Oct 30 18:36:38.807255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807297    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92981ab7-a41b-482f-98c3-a328e5e8d6f0" volumeName="kubernetes.io/empty-dir/92981ab7-a41b-482f-98c3-a328e5e8d6f0-ca-trust-extracted" seLinuxMountContext=""
Oct 30 18:36:38.809670 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807309    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92981ab7-a41b-482f-98c3-a328e5e8d6f0" volumeName="kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-bound-sa-token" seLinuxMountContext=""
Oct 30 18:36:38.809670 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807319    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b94a90ae-e06a-4e60-a48e-b567ce6e4edd" volumeName="kubernetes.io/configmap/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-config" seLinuxMountContext=""
Oct 30 18:36:38.809670 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807330    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9849d01-613f-439d-86de-b517ee4a3341" volumeName="kubernetes.io/configmap/b9849d01-613f-439d-86de-b517ee4a3341-serviceca" seLinuxMountContext=""
Oct 30 18:36:38.809670 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807341    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1c10266-f60d-4b43-b2ee-cef69669bc7b" volumeName="kubernetes.io/configmap/e1c10266-f60d-4b43-b2ee-cef69669bc7b-env-overrides" seLinuxMountContext=""
Oct 30 18:36:38.809670 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807353    4924 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e438abba-e5b6-41bb-bc67-1a1d322e6534" volumeName="kubernetes.io/projected/e438abba-e5b6-41bb-bc67-1a1d322e6534-kube-api-access-6hq7p" seLinuxMountContext=""
Oct 30 18:36:38.809670 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807363    4924 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 18:36:38.809670 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.807369    4924 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 18:36:38.816963 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.816942    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/1.log"
Oct 30 18:36:38.816963 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.816942    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/1.log"
Oct 30 18:36:38.822034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.821999    4924 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 18:36:38.826116 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.826081    4924 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 18:36:38.826116 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.826118    4924 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 18:36:38.826515 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.826500    4924 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 18:36:38.826576 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:38.826555    4924 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 18:36:38.848309 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.848282    4924 generic.go:356] "Generic (PLEG): container finished" podID="65beb5a5-1df3-42c8-9e14-769b4ad47255" containerID="af02f46d8eb4a9d0283a67ee6f6c588f2d9cb834b565def1ea574e349a30a56e" exitCode=0
Oct 30 18:36:38.848309 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.848307    4924 generic.go:356] "Generic (PLEG): container finished" podID="65beb5a5-1df3-42c8-9e14-769b4ad47255" containerID="1b14c6301e6f00ba59027cfce1260b9e8140986d05171a6dcb81c3ced71dfeb6" exitCode=0
Oct 30 18:36:38.848433 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.848314    4924 generic.go:356] "Generic (PLEG): container finished" podID="65beb5a5-1df3-42c8-9e14-769b4ad47255" containerID="ed3cf5ccf33456eff4cf9ddf778547bb8117b02ebfa3e208fd1c2c507e6244d3" exitCode=0
Oct 30 18:36:38.848433 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.848320    4924 generic.go:356] "Generic (PLEG): container finished" podID="65beb5a5-1df3-42c8-9e14-769b4ad47255" containerID="55d55076763e6a9a0ce5929024b3c0027fe2703b8d66edf482263c715dd6e4b8" exitCode=0
Oct 30 18:36:38.848433 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.848325    4924 generic.go:356] "Generic (PLEG): container finished" podID="65beb5a5-1df3-42c8-9e14-769b4ad47255" containerID="cc8930523543e9e5f841136ae664aa02bf6216f23f6cca9c4a6fdb9a7b6fc259" exitCode=0
Oct 30 18:36:38.848433 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.848331    4924 generic.go:356] "Generic (PLEG): container finished" podID="65beb5a5-1df3-42c8-9e14-769b4ad47255" containerID="ef49d038bacffd3b76c7e60e66730b18de394d2a31e9ae50b88f893ea54ed914" exitCode=0
Oct 30 18:36:38.874307 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.874279    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/1.log"
Oct 30 18:36:38.874307 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.874309    4924 generic.go:356] "Generic (PLEG): container finished" podID="60e0e397-bb2c-4637-9376-c5af871666cb" containerID="e7bca663a98eaf8619e4d44e2ab3f7281662e2728ad5d0f643a4a85b0146af10" exitCode=255
Oct 30 18:36:38.880737 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.880706    4924 generic.go:356] "Generic (PLEG): container finished" podID="0bdd5feb35ab75fe20b4acd57c05f061" containerID="9f10cc8efd5fe47ce08be827a22667753c7145bd11db93cfc05a9814b0c04f67" exitCode=0
Oct 30 18:36:38.896298 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.896242    4924 manager.go:324] Recovery completed
Oct 30 18:36:38.897203 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:38.897182    4924 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-224.us-west-2.compute.internal\" not found"
Oct 30 18:36:38.912742 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.912722    4924 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:38.915679 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.915663    4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:38.915751 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.915690    4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:38.915751 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.915700    4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:38.918075 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.918033    4924 cpu_manager.go:226] "Starting" policy="none"
Oct 30 18:36:38.918075 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.918050    4924 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 18:36:38.918223 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.918082    4924 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 18:36:38.918223 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.918153    4924 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:38.918316 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.918263    4924 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 18:36:38.918316 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.918279    4924 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 18:36:38.918316 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.918312    4924 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 18:36:38.918460 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.918322    4924 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 18:36:38.918460 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.918330    4924 policy_none.go:50] "Start"
Oct 30 18:36:38.918460 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.918350    4924 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 18:36:38.918460 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.918364    4924 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 18:36:38.918646 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.918590    4924 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 18:36:38.918646 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.918601    4924 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 18:36:38.918646 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.918610    4924 policy_none.go:44] "Start"
Oct 30 18:36:38.919380 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.919358    4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:38.919465 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.919428    4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:38.919465 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.919445    4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:38.923717 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.923688    4924 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 18:36:38.926543 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.926525    4924 manager.go:341] "Starting Device Plugin manager"
Oct 30 18:36:38.926618 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:38.926556    4924 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 18:36:38.926618 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.926568    4924 server.go:87] "Starting device plugin registration server"
Oct 30 18:36:38.926867 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.926851    4924 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 18:36:38.926938 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.926868    4924 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 18:36:38.927077 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.927049    4924 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 18:36:38.927187 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.927155    4924 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 18:36:38.927246 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.927194    4924 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 18:36:38.927246 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.927201    4924 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 18:36:38.927353 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.927277    4924 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal"]
Oct 30 18:36:38.927455 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.927440    4924 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:38.927740 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.927251    4924 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:36:38.927184877Z","UUID":"d001407e-a409-4e26-a6d7-5f26ae8186a7","Handler":null,"Name":"","Endpoint":""}
Oct 30 18:36:38.928973 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:38.928957    4924 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 18:36:38.929347 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:38.929331    4924 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-224.us-west-2.compute.internal\" not found"
Oct 30 18:36:38.929449 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.929424    4924 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 18:36:38.929562 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.929469    4924 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 18:36:38.929562 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.929521    4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:38.929562 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.929542    4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:38.929562 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.929556    4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:38.929989 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.929972    4924 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:38.930250 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.930235    4924 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:38.931791 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.931177    4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:38.931791 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.931211    4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:38.931791 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.931227    4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:38.931791 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.931227    4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:38.931791 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.931254    4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:38.931791 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.931266    4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:38.931791 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:38.931572    4924 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-224.us-west-2.compute.internal\" not found" node="ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:36:38.931791 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.931668    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal" event={"ID":"2e58124d8b2f46cf7f8af1e433d1d9a6","Type":"ContainerStarted","Data":"b71c61a669e677a3e50226c5b94e93d4c59bb83d095bba29d7e925ab298fe447"}
Oct 30 18:36:38.931791 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.931706    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal" event={"ID":"2e58124d8b2f46cf7f8af1e433d1d9a6","Type":"ContainerStarted","Data":"3c56149a6c66eaa82146cbbab520741d290fbb1283425fd10aa2409a3f28accf"}
Oct 30 18:36:38.931791 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.931743    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal" event={"ID":"0bdd5feb35ab75fe20b4acd57c05f061","Type":"ContainerStarted","Data":"27f84e9cd8fc24fcfa9e906f78856183d3215cbf13b6b115e616a208c1e7a1c8"}
Oct 30 18:36:38.931791 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.931754    4924 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:38.931791 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.931760    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal" event={"ID":"0bdd5feb35ab75fe20b4acd57c05f061","Type":"ContainerDied","Data":"9f10cc8efd5fe47ce08be827a22667753c7145bd11db93cfc05a9814b0c04f67"}
Oct 30 18:36:38.931791 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.931776    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal" event={"ID":"0bdd5feb35ab75fe20b4acd57c05f061","Type":"ContainerStarted","Data":"e4755db47b4602d74bc92410b4139a0c03ec31fe285b49cc74c99928d8351c06"}
Oct 30 18:36:38.932751 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.932729    4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:38.932751 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.932761    4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:38.932901 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.932776    4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:38.932951 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:38.932936    4924 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-224.us-west-2.compute.internal\" not found" node="ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:36:38.979221 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.979196    4924 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 18:36:38.979416 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.979402    4924 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:38.981040 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.981025    4924 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:38.981137 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.981122    4924 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:38.996465 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:38.996450    4924 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:36:39.027760 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.027743    4924 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:39.028409 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.028390    4924 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:39.028691 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.028678    4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:39.028738 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.028705    4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:39.028738 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.028716    4924 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-224.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:39.028792 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.028773    4924 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:36:39.054986 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.054964    4924 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:36:39.055082 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:39.055008    4924 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:36:39.055082 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.055019    4924 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:36:39.079382 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.079320    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e58124d8b2f46cf7f8af1e433d1d9a6-config\") pod \"kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal\" (UID: \"2e58124d8b2f46cf7f8af1e433d1d9a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:36:39.079469 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.079397    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0bdd5feb35ab75fe20b4acd57c05f061-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal\" (UID: \"0bdd5feb35ab75fe20b4acd57c05f061\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:36:39.079469 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.079422    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0bdd5feb35ab75fe20b4acd57c05f061-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal\" (UID: \"0bdd5feb35ab75fe20b4acd57c05f061\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:36:39.132754 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.132732    4924 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:36:39.132844 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.132791    4924 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:36:39.153586 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.153538    4924 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:36:39.153586 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:39.153572    4924 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:36:39.179720 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.179698    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e58124d8b2f46cf7f8af1e433d1d9a6-config\") pod \"kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal\" (UID: \"2e58124d8b2f46cf7f8af1e433d1d9a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:36:39.179824 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.179723    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0bdd5feb35ab75fe20b4acd57c05f061-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal\" (UID: \"0bdd5feb35ab75fe20b4acd57c05f061\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:36:39.179824 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.179744    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0bdd5feb35ab75fe20b4acd57c05f061-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal\" (UID: \"0bdd5feb35ab75fe20b4acd57c05f061\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:36:39.179824 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.179778    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0bdd5feb35ab75fe20b4acd57c05f061-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal\" (UID: \"0bdd5feb35ab75fe20b4acd57c05f061\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:36:39.179824 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.179783    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0bdd5feb35ab75fe20b4acd57c05f061-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal\" (UID: \"0bdd5feb35ab75fe20b4acd57c05f061\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:36:39.179824 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.179808    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e58124d8b2f46cf7f8af1e433d1d9a6-config\") pod \"kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal\" (UID: \"2e58124d8b2f46cf7f8af1e433d1d9a6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:36:39.783418 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.783393    4924 apiserver.go:52] "Watching apiserver"
Oct 30 18:36:39.887017 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.886992    4924 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:36:39.887188 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.887138    4924 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:36:39.929807 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.929781    4924 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 18:36:39.930418 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.930392    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-wpnq5","kube-system/kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal","openshift-console-operator/console-operator-7cd56b8dbc-jkgcq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz","openshift-cluster-node-tuning-operator/tuned-bhxxm","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal","openshift-network-console/networking-console-plugin-86c478bb88-6pktt","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v","kube-system/kas-connection-checker-685895c7c9-65rz2","openshift-network-diagnostics/network-check-source-6f78d59f99-fswdh","openshift-network-operator/iptables-alerter-44cfg","openshift-ovn-kubernetes/ovnkube-node-mgd4r","openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx","openshift-dns/node-resolver-cdf94","openshift-multus/multus-wd7km","openshift-image-registry/node-ca-vtp2j","kube-system/kas-connection-checker-685895c7c9-f4bz7","openshift-image-registry/image-registry-8598c87c5b-d4m59","openshift-dns/dns-default-sxk64","openshift-multus/network-metrics-daemon-sqkfq","kube-system/global-pull-secret-syncer-vs8lx","kube-system/konnectivity-agent-5v2r9","openshift-ingress/router-default-b44dd54dc-hqb7p","openshift-multus/multus-additional-cni-plugins-h44bp","kube-system/kas-connection-checker-685895c7c9-wl4tz","openshift-kube-storage-version-migrator/migrator-66df5d496b-dfv27","openshift-network-diagnostics/network-check-target-xcmps","openshift-image-registry/image-registry-7b777d677-gcvwt","openshift-insights/insights-operator-7c4986d449-776vb","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rpvxs","openshift-insights/insights-runtime-extractor-xcwnk","openshift-ingress-canary/ingress-canary-g6sxb"]
Oct 30 18:36:39.932036 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.932017    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:39.932356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.932343    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"
Oct 30 18:36:39.932862 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.932841    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:39.934577 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.934560    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:39.934894 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.934879    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6pktt"
Oct 30 18:36:39.938013 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.937996    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:39.944608 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.944588    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:39.945126 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.945114    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:39.946028 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.946008    4924 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:36:39.946126 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.946113    4924 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:36:39.946206 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:39.946189    4924 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:36:39.946753 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:39.946737    4924 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-224.us-west-2.compute.internal"
Oct 30 18:36:39.946931 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.946916    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:39.980648 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.980629    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 18:36:39.980648 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.980652    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 18:36:39.981422 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.981407    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 18:36:39.981883 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.981870    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 18:36:39.981946 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.981895    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 18:36:39.981982 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.981946    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 18:36:39.982155 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.982145    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 18:36:39.982198 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.982155    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 18:36:39.982198 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.982159    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 18:36:39.982260 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.982211    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 18:36:39.982396 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.982383    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 18:36:39.982724 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.982713    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 18:36:39.983546 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.983534    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 18:36:39.983594 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.983535    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 18:36:39.983594 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.983552    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 18:36:39.983679 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.983668    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-h758x\""
Oct 30 18:36:39.983848 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.983746    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-4cchw\""
Oct 30 18:36:39.994129 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.994113    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 18:36:39.994214 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.994167    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m6qzp\""
Oct 30 18:36:39.998810 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:39.998793    4924 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 18:36:40.024506 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.024484    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tlfn7\""
Oct 30 18:36:40.024571 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.024540    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cxkbs\""
Oct 30 18:36:40.024571 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.024542    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vpn2j\""
Oct 30 18:36:40.025317 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.025301    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h4btl\""
Oct 30 18:36:40.025495 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.025481    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vd777\""
Oct 30 18:36:40.025766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.025756    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 18:36:40.025895 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.025883    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vtx29\""
Oct 30 18:36:40.026222 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.026212    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 18:36:40.026578 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.026562    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 18:36:40.026651 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.026583    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j6hdp\""
Oct 30 18:36:40.026928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.026910    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9ph7g\""
Oct 30 18:36:40.026928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.026926    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 18:36:40.027051 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.026917    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 18:36:40.027051 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.026963    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7kzh2\""
Oct 30 18:36:40.027051 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.026971    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5slhp\""
Oct 30 18:36:40.027051 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.026974    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-swngw\""
Oct 30 18:36:40.027051 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.027016    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 18:36:40.027277 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.027060    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dnpxx\""
Oct 30 18:36:40.027277 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.027140    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mbxkm\""
Oct 30 18:36:40.027277 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.027271    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 18:36:40.027415 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.027281    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 18:36:40.027415 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.027366    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zmfwz\""
Oct 30 18:36:40.027482 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.027436    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5mz5g\""
Oct 30 18:36:40.081103 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.081032    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:40.081205 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.081129    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 18:36:40.081248 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.081209    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 18:36:40.081688 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.081671    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:40.082149 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.082128    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:40.082235 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.082135    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:40.082235 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.082167    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 18:36:40.082235 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.082217    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:40.082390 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.082306    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:40.082390 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.082338    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 18:36:40.082390 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.082370    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 18:36:40.082390 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.082378    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 18:36:40.082390 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.082382    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 18:36:40.082602 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.082407    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 18:36:40.082602 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.082592    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-bound-sa-token\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:40.082692 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.082619    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-cnibin\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.082692 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.082633    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 18:36:40.082692 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.082640    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-run-k8s-cni-cncf-io\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.082692 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.082665    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tsxg\" (UniqueName: \"kubernetes.io/projected/e1c10266-f60d-4b43-b2ee-cef69669bc7b-kube-api-access-4tsxg\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.082894 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.082715    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65beb5a5-1df3-42c8-9e14-769b4ad47255-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:36:40.082894 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.082754    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/74a85732-6555-48bf-9ece-4e863cd1fb93-multus-daemon-config\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.082894 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.082799    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkl9j\" (UniqueName: \"kubernetes.io/projected/74a85732-6555-48bf-9ece-4e863cd1fb93-kube-api-access-qkl9j\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.082894 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.082855    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 18:36:40.082894 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.082857    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-service-ca-bundle\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:40.082894 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.082865    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 18:36:40.082894 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.082894    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 18:36:40.083224 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.082934    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b9849d01-613f-439d-86de-b517ee4a3341-serviceca\") pod \"node-ca-vtp2j\" (UID: \"b9849d01-613f-439d-86de-b517ee4a3341\") " pod="openshift-image-registry/node-ca-vtp2j"
Oct 30 18:36:40.083224 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.082972    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb856\" (UniqueName: \"kubernetes.io/projected/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278-kube-api-access-vb856\") pod \"network-check-target-xcmps\" (UID: \"0b0aa783-d4ab-490c-a5f7-d7da5ad8d278\") " pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:36:40.083224 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083004    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-var-lib-cni-multus\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.083224 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083033    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12928ad7-4038-473a-a4aa-be1a6a42eabc-config-volume\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:40.083224 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083058    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f416f20b-a466-4f3d-bd2e-bb825b189067-host-slash\") pod \"iptables-alerter-44cfg\" (UID: \"f416f20b-a466-4f3d-bd2e-bb825b189067\") " pod="openshift-network-operator/iptables-alerter-44cfg"
Oct 30 18:36:40.083224 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083097    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-slash\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.083224 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083121    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1c10266-f60d-4b43-b2ee-cef69669bc7b-env-overrides\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.083224 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083145    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-kubelet-dir\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:36:40.083224 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083168    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-run-netns\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.083224 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083182    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 18:36:40.083224 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083196    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l74z8\" (UniqueName: \"kubernetes.io/projected/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-kube-api-access-l74z8\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:40.083224 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083203    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:40.083224 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083222    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-certificates\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083238    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083247    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92f38b64-26fb-42d6-826a-94cd37e8b668-original-pull-secret\") pod \"global-pull-secret-syncer-vs8lx\" (UID: \"92f38b64-26fb-42d6-826a-94cd37e8b668\") " pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083272    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a11d470f-d198-42ec-bee1-a900c4814473-data-volume\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083277    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083299    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-certificates\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083324    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b9849d01-613f-439d-86de-b517ee4a3341-serviceca\") pod \"node-ca-vtp2j\" (UID: \"b9849d01-613f-439d-86de-b517ee4a3341\") " pod="openshift-image-registry/node-ca-vtp2j"
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083334    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v526v\" (UniqueName: \"kubernetes.io/projected/60e0e397-bb2c-4637-9376-c5af871666cb-kube-api-access-v526v\") pod \"console-operator-7cd56b8dbc-jkgcq\" (UID: \"60e0e397-bb2c-4637-9376-c5af871666cb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083363    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-var-lib-openvswitch\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083401    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a11d470f-d198-42ec-bee1-a900c4814473-data-volume\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083448    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083458    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083478    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-certificates\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083521    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083570    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083588    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083627    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083635    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1c10266-f60d-4b43-b2ee-cef69669bc7b-env-overrides\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083676    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083693    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083728    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083759    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083757    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083774    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/92f38b64-26fb-42d6-826a-94cd37e8b668-original-pull-secret\") pod \"global-pull-secret-syncer-vs8lx\" (UID: \"92f38b64-26fb-42d6-826a-94cd37e8b668\") " pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083778    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-certificates\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083801    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083804    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 18:36:40.083818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083827    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-run-ovn\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083847    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083856    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083859    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12928ad7-4038-473a-a4aa-be1a6a42eabc-config-volume\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083869    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083873    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083864    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-system-cni-dir\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083699    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083924    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083916    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083956    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083970    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083959    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084000    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.083996    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-serving-cert\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084034    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084047    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084055    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084103    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084128    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084167    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-serving-cert\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084180    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-var-lib-kubelet\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084184    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084218    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6bwn\" (UniqueName: \"kubernetes.io/projected/3935b0fd-69be-4272-adc5-ecfbc7af04ae-kube-api-access-g6bwn\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084227    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084244    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60e0e397-bb2c-4637-9376-c5af871666cb-trusted-ca\") pod \"console-operator-7cd56b8dbc-jkgcq\" (UID: \"60e0e397-bb2c-4637-9376-c5af871666cb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084274    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084271    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmjgg\" (UniqueName: \"kubernetes.io/projected/05fe2f04-29f7-4999-87ef-b30035b34883-kube-api-access-mmjgg\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084335    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65beb5a5-1df3-42c8-9e14-769b4ad47255-cnibin\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084368    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be409e3c-3420-4250-bfa7-3b01ecd69d9b-image-registry-private-configuration\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084441    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084492    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-hostroot\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084528    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e954d56b-ace6-4539-a467-486c1e889886-config\") pod \"service-ca-operator-544d749c8c-wpnq5\" (UID: \"e954d56b-ace6-4539-a467-486c1e889886\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wpnq5"
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084611    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be409e3c-3420-4250-bfa7-3b01ecd69d9b-image-registry-private-configuration\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084650    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/92981ab7-a41b-482f-98c3-a328e5e8d6f0-ca-trust-extracted\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084671    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-sys\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084701    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e954d56b-ace6-4539-a467-486c1e889886-config\") pod \"service-ca-operator-544d749c8c-wpnq5\" (UID: \"e954d56b-ace6-4539-a467-486c1e889886\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wpnq5"
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084687    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084749    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xg8f\" (UniqueName: \"kubernetes.io/projected/5d05b776-9750-4c5b-8d91-4cf27329a1fc-kube-api-access-2xg8f\") pod \"volume-data-source-validator-58c9f5f46d-rpvxs\" (UID: \"5d05b776-9750-4c5b-8d91-4cf27329a1fc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rpvxs"
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084755    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/92981ab7-a41b-482f-98c3-a328e5e8d6f0-ca-trust-extracted\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084785    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhnjp\" (UniqueName: \"kubernetes.io/projected/90df09a3-8706-4883-b711-be63e8e9f191-kube-api-access-hhnjp\") pod \"node-resolver-cdf94\" (UID: \"90df09a3-8706-4883-b711-be63e8e9f191\") " pod="openshift-dns/node-resolver-cdf94"
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084803    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92f38b64-26fb-42d6-826a-94cd37e8b668-dbus\") pod \"global-pull-secret-syncer-vs8lx\" (UID: \"92f38b64-26fb-42d6-826a-94cd37e8b668\") " pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084921    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0648f559-00ac-4de0-8b9f-ce02d1b657a9-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk\" (UID: \"0648f559-00ac-4de0-8b9f-ce02d1b657a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk"
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084949    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/80e45018-1682-47cd-9523-dbeaf35a39cd-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-plkxx\" (UID: \"80e45018-1682-47cd-9523-dbeaf35a39cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:40.084943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.084974    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-run-systemd\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085004    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-cni-netd\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085040    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1c10266-f60d-4b43-b2ee-cef69669bc7b-ovnkube-config\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085093    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9z9l\" (UniqueName: \"kubernetes.io/projected/a11d470f-d198-42ec-bee1-a900c4814473-kube-api-access-v9z9l\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085108    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0648f559-00ac-4de0-8b9f-ce02d1b657a9-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk\" (UID: \"0648f559-00ac-4de0-8b9f-ce02d1b657a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085120    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dth6b\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-kube-api-access-dth6b\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085147    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e0da5dc-3a50-414f-99aa-0a8bbe303a31-agent-certs\") pod \"konnectivity-agent-5v2r9\" (UID: \"1e0da5dc-3a50-414f-99aa-0a8bbe303a31\") " pod="kube-system/konnectivity-agent-5v2r9"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085172    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e0da5dc-3a50-414f-99aa-0a8bbe303a31-konnectivity-ca\") pod \"konnectivity-agent-5v2r9\" (UID: \"1e0da5dc-3a50-414f-99aa-0a8bbe303a31\") " pod="kube-system/konnectivity-agent-5v2r9"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085199    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65beb5a5-1df3-42c8-9e14-769b4ad47255-system-cni-dir\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085248    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-cni-bin\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085265    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1c10266-f60d-4b43-b2ee-cef69669bc7b-ovnkube-config\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085274    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085298    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085326    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65beb5a5-1df3-42c8-9e14-769b4ad47255-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085353    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-socket-dir\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085399    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-multus-conf-dir\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085450    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-node-log\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085474    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-systemd\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085504    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65beb5a5-1df3-42c8-9e14-769b4ad47255-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085533    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdblb\" (UniqueName: \"kubernetes.io/projected/80e45018-1682-47cd-9523-dbeaf35a39cd-kube-api-access-vdblb\") pod \"cluster-monitoring-operator-884576c57-plkxx\" (UID: \"80e45018-1682-47cd-9523-dbeaf35a39cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085560    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90df09a3-8706-4883-b711-be63e8e9f191-hosts-file\") pod \"node-resolver-cdf94\" (UID: \"90df09a3-8706-4883-b711-be63e8e9f191\") " pod="openshift-dns/node-resolver-cdf94"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085585    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-etc-openvswitch\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085608    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-sysctl-conf\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085634    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-2xb8v\" (UID: \"b94a90ae-e06a-4e60-a48e-b567ce6e4edd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085661    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6pktt\" (UID: \"9b63e008-3071-4593-bba8-3bcc673b5dc2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6pktt"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085694    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-systemd-units\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085720    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks7sd\" (UniqueName: \"kubernetes.io/projected/21b6f672-2dc5-4523-9dfa-6dfaa30a8bac-kube-api-access-ks7sd\") pod \"ingress-canary-g6sxb\" (UID: \"21b6f672-2dc5-4523-9dfa-6dfaa30a8bac\") " pod="openshift-ingress-canary/ingress-canary-g6sxb"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085698    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65beb5a5-1df3-42c8-9e14-769b4ad47255-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085743    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-run-netns\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085767    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-var-lib-cni-bin\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085799    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-sys-fs\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085915    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e0da5dc-3a50-414f-99aa-0a8bbe303a31-konnectivity-ca\") pod \"konnectivity-agent-5v2r9\" (UID: \"1e0da5dc-3a50-414f-99aa-0a8bbe303a31\") " pod="kube-system/konnectivity-agent-5v2r9"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.085966    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65beb5a5-1df3-42c8-9e14-769b4ad47255-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086002    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-2xb8v\" (UID: \"b94a90ae-e06a-4e60-a48e-b567ce6e4edd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086047    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2xb8v\" (UID: \"b94a90ae-e06a-4e60-a48e-b567ce6e4edd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086058    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e0da5dc-3a50-414f-99aa-0a8bbe303a31-agent-certs\") pod \"konnectivity-agent-5v2r9\" (UID: \"1e0da5dc-3a50-414f-99aa-0a8bbe303a31\") " pod="kube-system/konnectivity-agent-5v2r9"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086098    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/92981ab7-a41b-482f-98c3-a328e5e8d6f0-installation-pull-secrets\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086194    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086229    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-lib-modules\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086262    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/92981ab7-a41b-482f-98c3-a328e5e8d6f0-installation-pull-secrets\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086276    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fdth\" (UniqueName: \"kubernetes.io/projected/6df0f1e7-07b7-4231-bc70-ba81aef29856-kube-api-access-4fdth\") pod \"network-metrics-daemon-sqkfq\" (UID: \"6df0f1e7-07b7-4231-bc70-ba81aef29856\") " pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086305    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr2vr\" (UniqueName: \"kubernetes.io/projected/9d148c95-78ae-43af-8942-460853970292-kube-api-access-cr2vr\") pod \"migrator-66df5d496b-dfv27\" (UID: \"9d148c95-78ae-43af-8942-460853970292\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dfv27"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086418    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-snapshots\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086447    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-multus-socket-dir-parent\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086490    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-snapshots\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086511    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-run-ovn-kubernetes\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086550    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dh8q\" (UniqueName: \"kubernetes.io/projected/a20a00eb-cde2-4df4-a191-0502fcaf288d-kube-api-access-5dh8q\") pod \"kas-connection-checker-685895c7c9-65rz2\" (UID: \"a20a00eb-cde2-4df4-a191-0502fcaf288d\") " pod="kube-system/kas-connection-checker-685895c7c9-65rz2"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086581    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-default-certificate\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086719    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-etc-selinux\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086749    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-multus-cni-dir\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086758    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-default-certificate\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:40.086766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086766    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086852    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-stats-auth\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086884    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpqmj\" (UniqueName: \"kubernetes.io/projected/65beb5a5-1df3-42c8-9e14-769b4ad47255-kube-api-access-jpqmj\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086900    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be409e3c-3420-4250-bfa7-3b01ecd69d9b-trusted-ca\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086915    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12928ad7-4038-473a-a4aa-be1a6a42eabc-tmp-dir\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086931    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086946    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-host\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086960    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1c10266-f60d-4b43-b2ee-cef69669bc7b-ovnkube-script-lib\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086982    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qznzq\" (UniqueName: \"kubernetes.io/projected/e954d56b-ace6-4539-a467-486c1e889886-kube-api-access-qznzq\") pod \"service-ca-operator-544d749c8c-wpnq5\" (UID: \"e954d56b-ace6-4539-a467-486c1e889886\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wpnq5"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.086999    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvl8c\" (UniqueName: \"kubernetes.io/projected/b9849d01-613f-439d-86de-b517ee4a3341-kube-api-access-jvl8c\") pod \"node-ca-vtp2j\" (UID: \"b9849d01-613f-439d-86de-b517ee4a3341\") " pod="openshift-image-registry/node-ca-vtp2j"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.087023    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12928ad7-4038-473a-a4aa-be1a6a42eabc-tmp-dir\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.087091    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-registration-dir\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.087365    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcz45\" (UniqueName: \"kubernetes.io/projected/12928ad7-4038-473a-a4aa-be1a6a42eabc-kube-api-access-qcz45\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.087378    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-stats-auth\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.087415    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74a85732-6555-48bf-9ece-4e863cd1fb93-cni-binary-copy\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.087584    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hp67\" (UniqueName: \"kubernetes.io/projected/d8842256-47fc-4dc5-91f6-65938f410080-kube-api-access-5hp67\") pod \"kas-connection-checker-685895c7c9-wl4tz\" (UID: \"d8842256-47fc-4dc5-91f6-65938f410080\") " pod="kube-system/kas-connection-checker-685895c7c9-wl4tz"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.087617    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx8hm\" (UniqueName: \"kubernetes.io/projected/5ef02004-71fc-4464-9550-bd094d27f21f-kube-api-access-dx8hm\") pod \"network-check-source-6f78d59f99-fswdh\" (UID: \"5ef02004-71fc-4464-9550-bd094d27f21f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fswdh"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.087652    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-device-dir\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.087665    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1c10266-f60d-4b43-b2ee-cef69669bc7b-ovnkube-script-lib\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.087686    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmg9f\" (UniqueName: \"kubernetes.io/projected/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-kube-api-access-dmg9f\") pod \"cluster-samples-operator-6ddb6c8b94-2xb8v\" (UID: \"b94a90ae-e06a-4e60-a48e-b567ce6e4edd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.087708    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-sysconfig\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.087863    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74a85732-6555-48bf-9ece-4e863cd1fb93-cni-binary-copy\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.087905    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9b63e008-3071-4593-bba8-3bcc673b5dc2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6pktt\" (UID: \"9b63e008-3071-4593-bba8-3bcc673b5dc2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6pktt"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.087996    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9b63e008-3071-4593-bba8-3bcc673b5dc2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6pktt\" (UID: \"9b63e008-3071-4593-bba8-3bcc673b5dc2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6pktt"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.088029    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hq7p\" (UniqueName: \"kubernetes.io/projected/e438abba-e5b6-41bb-bc67-1a1d322e6534-kube-api-access-6hq7p\") pod \"kas-connection-checker-685895c7c9-f4bz7\" (UID: \"e438abba-e5b6-41bb-bc67-1a1d322e6534\") " pod="kube-system/kas-connection-checker-685895c7c9-f4bz7"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.088083    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be409e3c-3420-4250-bfa7-3b01ecd69d9b-installation-pull-secrets\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.088115    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5c5k\" (UniqueName: \"kubernetes.io/projected/f416f20b-a466-4f3d-bd2e-bb825b189067-kube-api-access-m5c5k\") pod \"iptables-alerter-44cfg\" (UID: \"f416f20b-a466-4f3d-bd2e-bb825b189067\") " pod="openshift-network-operator/iptables-alerter-44cfg"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.088148    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-log-socket\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.088174    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a11d470f-d198-42ec-bee1-a900c4814473-crio-socket\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.088203    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9849d01-613f-439d-86de-b517ee4a3341-host\") pod \"node-ca-vtp2j\" (UID: \"b9849d01-613f-439d-86de-b517ee4a3341\") " pod="openshift-image-registry/node-ca-vtp2j"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.088393    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-sysctl-d\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.088430    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3935b0fd-69be-4272-adc5-ecfbc7af04ae-tmp\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.088463    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-run-multus-certs\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.088473    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be409e3c-3420-4250-bfa7-3b01ecd69d9b-installation-pull-secrets\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.088493    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-kubelet\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.088552    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1c10266-f60d-4b43-b2ee-cef69669bc7b-ovn-node-metrics-cert\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.088568    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3935b0fd-69be-4272-adc5-ecfbc7af04ae-tmp\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.088602    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.088663    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-bound-sa-token\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.088708    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p6bb\" (UniqueName: \"kubernetes.io/projected/9092fa02-cc22-41a2-b4ee-070298ecab99-kube-api-access-4p6bb\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.088759    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-etc-kubernetes\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.088792    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92981ab7-a41b-482f-98c3-a328e5e8d6f0-trusted-ca\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.088818    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1c10266-f60d-4b43-b2ee-cef69669bc7b-ovn-node-metrics-cert\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.088836    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0648f559-00ac-4de0-8b9f-ce02d1b657a9-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk\" (UID: \"0648f559-00ac-4de0-8b9f-ce02d1b657a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.089005    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f416f20b-a466-4f3d-bd2e-bb825b189067-iptables-alerter-script\") pod \"iptables-alerter-44cfg\" (UID: \"f416f20b-a466-4f3d-bd2e-bb825b189067\") " pod="openshift-network-operator/iptables-alerter-44cfg"
Oct 30 18:36:40.089034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.089037    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0648f559-00ac-4de0-8b9f-ce02d1b657a9-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk\" (UID: \"0648f559-00ac-4de0-8b9f-ce02d1b657a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.089169    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-tmp\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.089199    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be409e3c-3420-4250-bfa7-3b01ecd69d9b-ca-trust-extracted\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.089262    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.089300    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-modprobe-d\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.089329    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-run\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.089360    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/60e0e397-bb2c-4637-9376-c5af871666cb-config\") pod \"console-operator-7cd56b8dbc-jkgcq\" (UID: \"60e0e397-bb2c-4637-9376-c5af871666cb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.089435    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90df09a3-8706-4883-b711-be63e8e9f191-tmp-dir\") pod \"node-resolver-cdf94\" (UID: \"90df09a3-8706-4883-b711-be63e8e9f191\") " pod="openshift-dns/node-resolver-cdf94"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.089461    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs\") pod \"network-metrics-daemon-sqkfq\" (UID: \"6df0f1e7-07b7-4231-bc70-ba81aef29856\") " pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.089494    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8ggz\" (UniqueName: \"kubernetes.io/projected/0648f559-00ac-4de0-8b9f-ce02d1b657a9-kube-api-access-k8ggz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk\" (UID: \"0648f559-00ac-4de0-8b9f-ce02d1b657a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.089526    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-plkxx\" (UID: \"80e45018-1682-47cd-9523-dbeaf35a39cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.089558    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21b6f672-2dc5-4523-9dfa-6dfaa30a8bac-cert\") pod \"ingress-canary-g6sxb\" (UID: \"21b6f672-2dc5-4523-9dfa-6dfaa30a8bac\") " pod="openshift-ingress-canary/ingress-canary-g6sxb"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.089589    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-var-lib-kubelet\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.089621    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60e0e397-bb2c-4637-9376-c5af871666cb-serving-cert\") pod \"console-operator-7cd56b8dbc-jkgcq\" (UID: \"60e0e397-bb2c-4637-9376-c5af871666cb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.089730    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-run-openvswitch\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.089770    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/92981ab7-a41b-482f-98c3-a328e5e8d6f0-image-registry-private-configuration\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.089802    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-tuned\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.089833    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knktl\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-kube-api-access-knktl\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.089864    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92f38b64-26fb-42d6-826a-94cd37e8b668-kubelet-config\") pod \"global-pull-secret-syncer-vs8lx\" (UID: \"92f38b64-26fb-42d6-826a-94cd37e8b668\") " pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.089890    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e954d56b-ace6-4539-a467-486c1e889886-serving-cert\") pod \"service-ca-operator-544d749c8c-wpnq5\" (UID: \"e954d56b-ace6-4539-a467-486c1e889886\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wpnq5"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.089919    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-kubernetes\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.089955    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65beb5a5-1df3-42c8-9e14-769b4ad47255-cni-binary-copy\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.090255    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65beb5a5-1df3-42c8-9e14-769b4ad47255-cni-binary-copy\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.090393    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60e0e397-bb2c-4637-9376-c5af871666cb-serving-cert\") pod \"console-operator-7cd56b8dbc-jkgcq\" (UID: \"60e0e397-bb2c-4637-9376-c5af871666cb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.090551    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/60e0e397-bb2c-4637-9376-c5af871666cb-config\") pod \"console-operator-7cd56b8dbc-jkgcq\" (UID: \"60e0e397-bb2c-4637-9376-c5af871666cb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.090615    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be409e3c-3420-4250-bfa7-3b01ecd69d9b-ca-trust-extracted\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.090833    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f416f20b-a466-4f3d-bd2e-bb825b189067-iptables-alerter-script\") pod \"iptables-alerter-44cfg\" (UID: \"f416f20b-a466-4f3d-bd2e-bb825b189067\") " pod="openshift-network-operator/iptables-alerter-44cfg"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.090916    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-tmp\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:40.091002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.090979    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90df09a3-8706-4883-b711-be63e8e9f191-tmp-dir\") pod \"node-resolver-cdf94\" (UID: \"90df09a3-8706-4883-b711-be63e8e9f191\") " pod="openshift-dns/node-resolver-cdf94"
Oct 30 18:36:40.092324 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.091213    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/92981ab7-a41b-482f-98c3-a328e5e8d6f0-image-registry-private-configuration\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:40.092324 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.091279    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-tuned\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.092324 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.091438    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e954d56b-ace6-4539-a467-486c1e889886-serving-cert\") pod \"service-ca-operator-544d749c8c-wpnq5\" (UID: \"e954d56b-ace6-4539-a467-486c1e889886\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wpnq5"
Oct 30 18:36:40.092324 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.091668    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21b6f672-2dc5-4523-9dfa-6dfaa30a8bac-cert\") pod \"ingress-canary-g6sxb\" (UID: \"21b6f672-2dc5-4523-9dfa-6dfaa30a8bac\") " pod="openshift-ingress-canary/ingress-canary-g6sxb"
Oct 30 18:36:40.093526 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.093507    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-service-ca-bundle\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:40.093592 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.093554    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/74a85732-6555-48bf-9ece-4e863cd1fb93-multus-daemon-config\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.128749 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.128729    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 18:36:40.131188 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.131169    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 18:36:40.131268 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.131212    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 18:36:40.131736 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.131722    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 18:36:40.134604 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.134588    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/60e0e397-bb2c-4637-9376-c5af871666cb-trusted-ca\") pod \"console-operator-7cd56b8dbc-jkgcq\" (UID: \"60e0e397-bb2c-4637-9376-c5af871666cb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:40.135680 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.135665    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/80e45018-1682-47cd-9523-dbeaf35a39cd-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-plkxx\" (UID: \"80e45018-1682-47cd-9523-dbeaf35a39cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:40.137588 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.137569    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:40.137668 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.137598    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be409e3c-3420-4250-bfa7-3b01ecd69d9b-trusted-ca\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:40.138210 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.138189    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-bound-sa-token\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:40.138338 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.138325    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkl9j\" (UniqueName: \"kubernetes.io/projected/74a85732-6555-48bf-9ece-4e863cd1fb93-kube-api-access-qkl9j\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.138414 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.138400    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb856\" (UniqueName: \"kubernetes.io/projected/0b0aa783-d4ab-490c-a5f7-d7da5ad8d278-kube-api-access-vb856\") pod \"network-check-target-xcmps\" (UID: \"0b0aa783-d4ab-490c-a5f7-d7da5ad8d278\") " pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:36:40.139439 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.139418    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92981ab7-a41b-482f-98c3-a328e5e8d6f0-trusted-ca\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:40.139538 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.139522    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tsxg\" (UniqueName: \"kubernetes.io/projected/e1c10266-f60d-4b43-b2ee-cef69669bc7b-kube-api-access-4tsxg\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.146094 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.146060    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdblb\" (UniqueName: \"kubernetes.io/projected/80e45018-1682-47cd-9523-dbeaf35a39cd-kube-api-access-vdblb\") pod \"cluster-monitoring-operator-884576c57-plkxx\" (UID: \"80e45018-1682-47cd-9523-dbeaf35a39cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:40.146285 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.146267    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks7sd\" (UniqueName: \"kubernetes.io/projected/21b6f672-2dc5-4523-9dfa-6dfaa30a8bac-kube-api-access-ks7sd\") pod \"ingress-canary-g6sxb\" (UID: \"21b6f672-2dc5-4523-9dfa-6dfaa30a8bac\") " pod="openshift-ingress-canary/ingress-canary-g6sxb"
Oct 30 18:36:40.146901 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.146663    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhnjp\" (UniqueName: \"kubernetes.io/projected/90df09a3-8706-4883-b711-be63e8e9f191-kube-api-access-hhnjp\") pod \"node-resolver-cdf94\" (UID: \"90df09a3-8706-4883-b711-be63e8e9f191\") " pod="openshift-dns/node-resolver-cdf94"
Oct 30 18:36:40.146901 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.146709    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dh8q\" (UniqueName: \"kubernetes.io/projected/a20a00eb-cde2-4df4-a191-0502fcaf288d-kube-api-access-5dh8q\") pod \"kas-connection-checker-685895c7c9-65rz2\" (UID: \"a20a00eb-cde2-4df4-a191-0502fcaf288d\") " pod="kube-system/kas-connection-checker-685895c7c9-65rz2"
Oct 30 18:36:40.146901 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.146808    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l74z8\" (UniqueName: \"kubernetes.io/projected/abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9-kube-api-access-l74z8\") pod \"insights-operator-7c4986d449-776vb\" (UID: \"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9\") " pod="openshift-insights/insights-operator-7c4986d449-776vb"
Oct 30 18:36:40.146901 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.146848    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9z9l\" (UniqueName: \"kubernetes.io/projected/a11d470f-d198-42ec-bee1-a900c4814473-kube-api-access-v9z9l\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:40.146901 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.146862    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hq7p\" (UniqueName: \"kubernetes.io/projected/e438abba-e5b6-41bb-bc67-1a1d322e6534-kube-api-access-6hq7p\") pod \"kas-connection-checker-685895c7c9-f4bz7\" (UID: \"e438abba-e5b6-41bb-bc67-1a1d322e6534\") " pod="kube-system/kas-connection-checker-685895c7c9-f4bz7"
Oct 30 18:36:40.147156 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.146904    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6bwn\" (UniqueName: \"kubernetes.io/projected/3935b0fd-69be-4272-adc5-ecfbc7af04ae-kube-api-access-g6bwn\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.147156 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.147030    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xg8f\" (UniqueName: \"kubernetes.io/projected/5d05b776-9750-4c5b-8d91-4cf27329a1fc-kube-api-access-2xg8f\") pod \"volume-data-source-validator-58c9f5f46d-rpvxs\" (UID: \"5d05b776-9750-4c5b-8d91-4cf27329a1fc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rpvxs"
Oct 30 18:36:40.147156 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.147091    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v526v\" (UniqueName: \"kubernetes.io/projected/60e0e397-bb2c-4637-9376-c5af871666cb-kube-api-access-v526v\") pod \"console-operator-7cd56b8dbc-jkgcq\" (UID: \"60e0e397-bb2c-4637-9376-c5af871666cb\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:40.147156 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.147098    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr2vr\" (UniqueName: \"kubernetes.io/projected/9d148c95-78ae-43af-8942-460853970292-kube-api-access-cr2vr\") pod \"migrator-66df5d496b-dfv27\" (UID: \"9d148c95-78ae-43af-8942-460853970292\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dfv27"
Oct 30 18:36:40.148181 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.148167    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dth6b\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-kube-api-access-dth6b\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:40.148539 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.148515    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-bound-sa-token\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:40.148646 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.148579    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmjgg\" (UniqueName: \"kubernetes.io/projected/05fe2f04-29f7-4999-87ef-b30035b34883-kube-api-access-mmjgg\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:40.148758 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.148741    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fdth\" (UniqueName: \"kubernetes.io/projected/6df0f1e7-07b7-4231-bc70-ba81aef29856-kube-api-access-4fdth\") pod \"network-metrics-daemon-sqkfq\" (UID: \"6df0f1e7-07b7-4231-bc70-ba81aef29856\") " pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:40.148986 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.148971    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8ggz\" (UniqueName: \"kubernetes.io/projected/0648f559-00ac-4de0-8b9f-ce02d1b657a9-kube-api-access-k8ggz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk\" (UID: \"0648f559-00ac-4de0-8b9f-ce02d1b657a9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk"
Oct 30 18:36:40.149119 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.149097    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knktl\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-kube-api-access-knktl\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:40.149119 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.149112    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcz45\" (UniqueName: \"kubernetes.io/projected/12928ad7-4038-473a-a4aa-be1a6a42eabc-kube-api-access-qcz45\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:40.149233 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.149103    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvl8c\" (UniqueName: \"kubernetes.io/projected/b9849d01-613f-439d-86de-b517ee4a3341-kube-api-access-jvl8c\") pod \"node-ca-vtp2j\" (UID: \"b9849d01-613f-439d-86de-b517ee4a3341\") " pod="openshift-image-registry/node-ca-vtp2j"
Oct 30 18:36:40.149666 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.149644    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p6bb\" (UniqueName: \"kubernetes.io/projected/9092fa02-cc22-41a2-b4ee-070298ecab99-kube-api-access-4p6bb\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:36:40.149768 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.149753    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpqmj\" (UniqueName: \"kubernetes.io/projected/65beb5a5-1df3-42c8-9e14-769b4ad47255-kube-api-access-jpqmj\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:36:40.150520 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.150498    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx8hm\" (UniqueName: \"kubernetes.io/projected/5ef02004-71fc-4464-9550-bd094d27f21f-kube-api-access-dx8hm\") pod \"network-check-source-6f78d59f99-fswdh\" (UID: \"5ef02004-71fc-4464-9550-bd094d27f21f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fswdh"
Oct 30 18:36:40.150579 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.150517    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmg9f\" (UniqueName: \"kubernetes.io/projected/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-kube-api-access-dmg9f\") pod \"cluster-samples-operator-6ddb6c8b94-2xb8v\" (UID: \"b94a90ae-e06a-4e60-a48e-b567ce6e4edd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"
Oct 30 18:36:40.150789 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.150776    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qznzq\" (UniqueName: \"kubernetes.io/projected/e954d56b-ace6-4539-a467-486c1e889886-kube-api-access-qznzq\") pod \"service-ca-operator-544d749c8c-wpnq5\" (UID: \"e954d56b-ace6-4539-a467-486c1e889886\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wpnq5"
Oct 30 18:36:40.150820 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.150777    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hp67\" (UniqueName: \"kubernetes.io/projected/d8842256-47fc-4dc5-91f6-65938f410080-kube-api-access-5hp67\") pod \"kas-connection-checker-685895c7c9-wl4tz\" (UID: \"d8842256-47fc-4dc5-91f6-65938f410080\") " pod="kube-system/kas-connection-checker-685895c7c9-wl4tz"
Oct 30 18:36:40.150920 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.150906    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5c5k\" (UniqueName: \"kubernetes.io/projected/f416f20b-a466-4f3d-bd2e-bb825b189067-kube-api-access-m5c5k\") pod \"iptables-alerter-44cfg\" (UID: \"f416f20b-a466-4f3d-bd2e-bb825b189067\") " pod="openshift-network-operator/iptables-alerter-44cfg"
Oct 30 18:36:40.190360 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190340    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-host\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.190489 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190366    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-registration-dir\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:36:40.190489 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190390    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-device-dir\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:36:40.190489 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190413    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-sysconfig\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.190489 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190468    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-device-dir\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:36:40.190680 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190463    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-host\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.190680 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190570    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-log-socket\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.190680 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190592    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-registration-dir\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:36:40.190680 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190597    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a11d470f-d198-42ec-bee1-a900c4814473-crio-socket\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:40.190680 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190635    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a11d470f-d198-42ec-bee1-a900c4814473-crio-socket\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:40.190680 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190648    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9849d01-613f-439d-86de-b517ee4a3341-host\") pod \"node-ca-vtp2j\" (UID: \"b9849d01-613f-439d-86de-b517ee4a3341\") " pod="openshift-image-registry/node-ca-vtp2j"
Oct 30 18:36:40.190680 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190678    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-sysctl-d\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.190963 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190707    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-run-multus-certs\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.190963 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190722    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-sysconfig\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.190963 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190722    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-kubelet\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.190963 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190752    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-kubelet\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.190963 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190682    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b9849d01-613f-439d-86de-b517ee4a3341-host\") pod \"node-ca-vtp2j\" (UID: \"b9849d01-613f-439d-86de-b517ee4a3341\") " pod="openshift-image-registry/node-ca-vtp2j"
Oct 30 18:36:40.190963 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190767    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:40.190963 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190784    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-run-multus-certs\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.190963 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190786    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-log-socket\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.190963 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.190813    4924 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:40.190963 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190830    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-sysctl-d\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.190963 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190838    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-etc-kubernetes\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.190963 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.190873    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm podName:a11d470f-d198-42ec-bee1-a900c4814473 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:40.690860082 +0000 UTC m=+1.967910533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xcwnk" (UID: "a11d470f-d198-42ec-bee1-a900c4814473") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:40.190963 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190810    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-etc-kubernetes\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.190963 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190908    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:40.190963 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190927    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-modprobe-d\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.190963 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190945    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-run\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.190963 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190959    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs\") pod \"network-metrics-daemon-sqkfq\" (UID: \"6df0f1e7-07b7-4231-bc70-ba81aef29856\") " pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.190982    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-plkxx\" (UID: \"80e45018-1682-47cd-9523-dbeaf35a39cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.190999    4924 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191008    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-var-lib-kubelet\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191034    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-run-openvswitch\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.191014    4924 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8598c87c5b-d4m59: secret "image-registry-tls" not found
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191060    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92f38b64-26fb-42d6-826a-94cd37e8b668-kubelet-config\") pod \"global-pull-secret-syncer-vs8lx\" (UID: \"92f38b64-26fb-42d6-826a-94cd37e8b668\") " pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.191081    4924 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191102    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-kubernetes\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191115    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-modprobe-d\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191037    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-run\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191126    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-cnibin\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191120    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-run-openvswitch\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.191083    4924 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.191145    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls podName:92981ab7-a41b-482f-98c3-a328e5e8d6f0 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:40.691131197 +0000 UTC m=+1.968181645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls") pod "image-registry-8598c87c5b-d4m59" (UID: "92981ab7-a41b-482f-98c3-a328e5e8d6f0") : secret "image-registry-tls" not found
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191076    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-var-lib-kubelet\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191167    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/92f38b64-26fb-42d6-826a-94cd37e8b668-kubelet-config\") pod \"global-pull-secret-syncer-vs8lx\" (UID: \"92f38b64-26fb-42d6-826a-94cd37e8b668\") " pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191168    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-kubernetes\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.191189    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs podName:6df0f1e7-07b7-4231-bc70-ba81aef29856 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:40.691179232 +0000 UTC m=+1.968229655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs") pod "network-metrics-daemon-sqkfq" (UID: "6df0f1e7-07b7-4231-bc70-ba81aef29856") : secret "metrics-daemon-secret" not found
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.191207    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls podName:80e45018-1682-47cd-9523-dbeaf35a39cd nodeName:}" failed. No retries permitted until 2026-10-30 18:36:40.691201797 +0000 UTC m=+1.968252220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-plkxx" (UID: "80e45018-1682-47cd-9523-dbeaf35a39cd") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191181    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-cnibin\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191219    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-run-k8s-cni-cncf-io\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191252    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-run-k8s-cni-cncf-io\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191256    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65beb5a5-1df3-42c8-9e14-769b4ad47255-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191289    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-var-lib-cni-multus\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191316    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f416f20b-a466-4f3d-bd2e-bb825b189067-host-slash\") pod \"iptables-alerter-44cfg\" (UID: \"f416f20b-a466-4f3d-bd2e-bb825b189067\") " pod="openshift-network-operator/iptables-alerter-44cfg"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191335    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65beb5a5-1df3-42c8-9e14-769b4ad47255-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191338    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-var-lib-cni-multus\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191337    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-slash\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191375    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f416f20b-a466-4f3d-bd2e-bb825b189067-host-slash\") pod \"iptables-alerter-44cfg\" (UID: \"f416f20b-a466-4f3d-bd2e-bb825b189067\") " pod="openshift-network-operator/iptables-alerter-44cfg"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191375    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-slash\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191388    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-kubelet-dir\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191415    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-kubelet-dir\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191420    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-run-netns\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191450    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-var-lib-openvswitch\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191465    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-run-netns\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191474    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-run-ovn\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191499    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-system-cni-dir\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191520    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191529    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-var-lib-openvswitch\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191537    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-run-ovn\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191543    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-var-lib-kubelet\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191552    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-system-cni-dir\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191568    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65beb5a5-1df3-42c8-9e14-769b4ad47255-cnibin\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.191593    4924 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191623    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-hostroot\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.191636    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls podName:12928ad7-4038-473a-a4aa-be1a6a42eabc nodeName:}" failed. No retries permitted until 2026-10-30 18:36:40.691627757 +0000 UTC m=+1.968678193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls") pod "dns-default-sxk64" (UID: "12928ad7-4038-473a-a4aa-be1a6a42eabc") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191592    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-hostroot\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191641    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-var-lib-kubelet\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191645    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65beb5a5-1df3-42c8-9e14-769b4ad47255-cnibin\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191664    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-sys\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191700    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191727    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92f38b64-26fb-42d6-826a-94cd37e8b668-dbus\") pod \"global-pull-secret-syncer-vs8lx\" (UID: \"92f38b64-26fb-42d6-826a-94cd37e8b668\") " pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191734    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-sys\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191752    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-run-systemd\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191774    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-cni-netd\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191792    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/92f38b64-26fb-42d6-826a-94cd37e8b668-dbus\") pod \"global-pull-secret-syncer-vs8lx\" (UID: \"92f38b64-26fb-42d6-826a-94cd37e8b668\") " pod="kube-system/global-pull-secret-syncer-vs8lx"
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.191800    4924 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:40.191772 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.191814    4924 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b777d677-gcvwt: secret "image-registry-tls" not found
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191818    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-cni-netd\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191800    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-run-systemd\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191831    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65beb5a5-1df3-42c8-9e14-769b4ad47255-system-cni-dir\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191802    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65beb5a5-1df3-42c8-9e14-769b4ad47255-system-cni-dir\") pod \"multus-additional-cni-plugins-h44bp\" (UID: \"65beb5a5-1df3-42c8-9e14-769b4ad47255\") " pod="openshift-multus/multus-additional-cni-plugins-h44bp"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.191843    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls podName:be409e3c-3420-4250-bfa7-3b01ecd69d9b nodeName:}" failed. No retries permitted until 2026-10-30 18:36:40.691834049 +0000 UTC m=+1.968884476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls") pod "image-registry-7b777d677-gcvwt" (UID: "be409e3c-3420-4250-bfa7-3b01ecd69d9b") : secret "image-registry-tls" not found
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191870    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-cni-bin\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191889    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191906    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191921    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-socket-dir\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191935    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-multus-conf-dir\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191942    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-cni-bin\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191949    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-node-log\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191966    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-systemd\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.191977    4924 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191985    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90df09a3-8706-4883-b711-be63e8e9f191-hosts-file\") pod \"node-resolver-cdf94\" (UID: \"90df09a3-8706-4883-b711-be63e8e9f191\") " pod="openshift-dns/node-resolver-cdf94"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191985    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-multus-conf-dir\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191990    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-node-log\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.191999    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-etc-openvswitch\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192010    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-socket-dir\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.192020    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs podName:05fe2f04-29f7-4999-87ef-b30035b34883 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:40.692006446 +0000 UTC m=+1.969056868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs") pod "router-default-b44dd54dc-hqb7p" (UID: "05fe2f04-29f7-4999-87ef-b30035b34883") : secret "router-metrics-certs-default" not found
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192023    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-systemd\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192031    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-etc-openvswitch\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192040    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-sysctl-conf\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192060    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6pktt\" (UID: \"9b63e008-3071-4593-bba8-3bcc673b5dc2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6pktt"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.192083    4924 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192095    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-systemd-units\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192118    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-run-netns\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192038    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90df09a3-8706-4883-b711-be63e8e9f191-hosts-file\") pod \"node-resolver-cdf94\" (UID: \"90df09a3-8706-4883-b711-be63e8e9f191\") " pod="openshift-dns/node-resolver-cdf94"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192144    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-var-lib-cni-bin\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192145    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-systemd-units\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.192156    4924 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.192164    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls podName:a11d470f-d198-42ec-bee1-a900c4814473 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:40.692152163 +0000 UTC m=+1.969202586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xcwnk" (UID: "a11d470f-d198-42ec-bee1-a900c4814473") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192163    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-etc-sysctl-conf\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192181    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-run-netns\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192186    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-host-var-lib-cni-bin\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.192198    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert podName:9b63e008-3071-4593-bba8-3bcc673b5dc2 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:40.692184614 +0000 UTC m=+1.969235054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6pktt" (UID: "9b63e008-3071-4593-bba8-3bcc673b5dc2") : secret "networking-console-plugin-cert" not found
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192226    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-sys-fs\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192244    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-sys-fs\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192245    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2xb8v\" (UID: \"b94a90ae-e06a-4e60-a48e-b567ce6e4edd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192280    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192297    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-lib-modules\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192315    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-multus-socket-dir-parent\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192327    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192331    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-run-ovn-kubernetes\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192349    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1c10266-f60d-4b43-b2ee-cef69669bc7b-host-run-ovn-kubernetes\") pod \"ovnkube-node-mgd4r\" (UID: \"e1c10266-f60d-4b43-b2ee-cef69669bc7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.192283    4924 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.192380    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls podName:b94a90ae-e06a-4e60-a48e-b567ce6e4edd nodeName:}" failed. No retries permitted until 2026-10-30 18:36:40.692372934 +0000 UTC m=+1.969423360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2xb8v" (UID: "b94a90ae-e06a-4e60-a48e-b567ce6e4edd") : secret "samples-operator-tls" not found
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192355    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-etc-selinux\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192388    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9092fa02-cc22-41a2-b4ee-070298ecab99-etc-selinux\") pod \"aws-ebs-csi-driver-node-55ndz\" (UID: \"9092fa02-cc22-41a2-b4ee-070298ecab99\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55ndz"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192396    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-multus-socket-dir-parent\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192401    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-multus-cni-dir\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192423    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3935b0fd-69be-4272-adc5-ecfbc7af04ae-lib-modules\") pod \"tuned-bhxxm\" (UID: \"3935b0fd-69be-4272-adc5-ecfbc7af04ae\") " pod="openshift-cluster-node-tuning-operator/tuned-bhxxm"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192435    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74a85732-6555-48bf-9ece-4e863cd1fb93-multus-cni-dir\") pod \"multus-wd7km\" (UID: \"74a85732-6555-48bf-9ece-4e863cd1fb93\") " pod="openshift-multus/multus-wd7km"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.192436    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:40.193593 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.192560    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle podName:05fe2f04-29f7-4999-87ef-b30035b34883 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:40.692545407 +0000 UTC m=+1.969595847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle") pod "router-default-b44dd54dc-hqb7p" (UID: "05fe2f04-29f7-4999-87ef-b30035b34883") : configmap references non-existent config key: service-ca.crt
Oct 30 18:36:40.234232 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.234205    4924 scope.go:122] "RemoveContainer" containerID="e7bca663a98eaf8619e4d44e2ab3f7281662e2728ad5d0f643a4a85b0146af10"
Oct 30 18:36:40.664844 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.664760    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:40.697288 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.697261    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:40.697393 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.697294    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:40.697393 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.697330    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6pktt\" (UID: \"9b63e008-3071-4593-bba8-3bcc673b5dc2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6pktt"
Oct 30 18:36:40.697393 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.697367    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2xb8v\" (UID: \"b94a90ae-e06a-4e60-a48e-b567ce6e4edd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"
Oct 30 18:36:40.697552 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.697407    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:40.697552 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.697413    4924 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:40.697552 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.697445    4924 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:36:40.697552 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.697458    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:40.697552 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.697470    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls podName:a11d470f-d198-42ec-bee1-a900c4814473 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:41.697453612 +0000 UTC m=+2.974504053 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xcwnk" (UID: "a11d470f-d198-42ec-bee1-a900c4814473") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:40.697552 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.697482    4924 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:36:40.697552 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.697504    4924 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:40.697552 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.697506    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert podName:9b63e008-3071-4593-bba8-3bcc673b5dc2 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:41.697488976 +0000 UTC m=+2.974539402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6pktt" (UID: "9b63e008-3071-4593-bba8-3bcc673b5dc2") : secret "networking-console-plugin-cert" not found
Oct 30 18:36:40.697552 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.697534    4924 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:36:40.697552 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.697548    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:40.697552 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.697558    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle podName:05fe2f04-29f7-4999-87ef-b30035b34883 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:41.697549525 +0000 UTC m=+2.974599969 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle") pod "router-default-b44dd54dc-hqb7p" (UID: "05fe2f04-29f7-4999-87ef-b30035b34883") : configmap references non-existent config key: service-ca.crt
Oct 30 18:36:40.697918 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.697595    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs podName:05fe2f04-29f7-4999-87ef-b30035b34883 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:41.697583667 +0000 UTC m=+2.974634104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs") pod "router-default-b44dd54dc-hqb7p" (UID: "05fe2f04-29f7-4999-87ef-b30035b34883") : secret "router-metrics-certs-default" not found
Oct 30 18:36:40.697918 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.697605    4924 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:40.697918 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.697615    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm podName:a11d470f-d198-42ec-bee1-a900c4814473 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:41.697605791 +0000 UTC m=+2.974656219 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xcwnk" (UID: "a11d470f-d198-42ec-bee1-a900c4814473") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:40.697918 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.697617    4924 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8598c87c5b-d4m59: secret "image-registry-tls" not found
Oct 30 18:36:40.697918 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.697632    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls podName:b94a90ae-e06a-4e60-a48e-b567ce6e4edd nodeName:}" failed. No retries permitted until 2026-10-30 18:36:41.697623875 +0000 UTC m=+2.974674299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2xb8v" (UID: "b94a90ae-e06a-4e60-a48e-b567ce6e4edd") : secret "samples-operator-tls" not found
Oct 30 18:36:40.697918 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.697656    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls podName:92981ab7-a41b-482f-98c3-a328e5e8d6f0 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:41.697649434 +0000 UTC m=+2.974699857 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls") pod "image-registry-8598c87c5b-d4m59" (UID: "92981ab7-a41b-482f-98c3-a328e5e8d6f0") : secret "image-registry-tls" not found
Oct 30 18:36:40.697918 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.697669    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs\") pod \"network-metrics-daemon-sqkfq\" (UID: \"6df0f1e7-07b7-4231-bc70-ba81aef29856\") " pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:40.697918 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.697688    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-plkxx\" (UID: \"80e45018-1682-47cd-9523-dbeaf35a39cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:40.697918 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.697736    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:40.697918 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.697745    4924 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:36:40.697918 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.697771    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs podName:6df0f1e7-07b7-4231-bc70-ba81aef29856 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:41.697764367 +0000 UTC m=+2.974814789 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs") pod "network-metrics-daemon-sqkfq" (UID: "6df0f1e7-07b7-4231-bc70-ba81aef29856") : secret "metrics-daemon-secret" not found
Oct 30 18:36:40.697918 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.697759    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:40.697918 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.697823    4924 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:40.697918 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.697827    4924 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:40.697918 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.697837    4924 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b777d677-gcvwt: secret "image-registry-tls" not found
Oct 30 18:36:40.697918 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.697844    4924 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:40.697918 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.697864    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls podName:80e45018-1682-47cd-9523-dbeaf35a39cd nodeName:}" failed. No retries permitted until 2026-10-30 18:36:41.697852272 +0000 UTC m=+2.974902695 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-plkxx" (UID: "80e45018-1682-47cd-9523-dbeaf35a39cd") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:40.697918 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.697877    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls podName:be409e3c-3420-4250-bfa7-3b01ecd69d9b nodeName:}" failed. No retries permitted until 2026-10-30 18:36:41.697871328 +0000 UTC m=+2.974921751 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls") pod "image-registry-7b777d677-gcvwt" (UID: "be409e3c-3420-4250-bfa7-3b01ecd69d9b") : secret "image-registry-tls" not found
Oct 30 18:36:40.697918 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.697887    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls podName:12928ad7-4038-473a-a4aa-be1a6a42eabc nodeName:}" failed. No retries permitted until 2026-10-30 18:36:41.697881337 +0000 UTC m=+2.974931760 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls") pod "dns-default-sxk64" (UID: "12928ad7-4038-473a-a4aa-be1a6a42eabc") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:40.890666 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.890640    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 18:36:40.891043 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.891026    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/1.log"
Oct 30 18:36:40.891101 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.891054    4924 generic.go:356] "Generic (PLEG): container finished" podID="60e0e397-bb2c-4637-9376-c5af871666cb" containerID="6018391a13dc4aa436c44d8661b8b08ebef2326b51aeb46d220860f009c99288" exitCode=255
Oct 30 18:36:40.891159 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.891133    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq" event={"ID":"60e0e397-bb2c-4637-9376-c5af871666cb","Type":"ContainerDied","Data":"6018391a13dc4aa436c44d8661b8b08ebef2326b51aeb46d220860f009c99288"}
Oct 30 18:36:40.891214 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.891201    4924 scope.go:122] "RemoveContainer" containerID="e7bca663a98eaf8619e4d44e2ab3f7281662e2728ad5d0f643a4a85b0146af10"
Oct 30 18:36:40.891618 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:40.891601    4924 scope.go:122] "RemoveContainer" containerID="6018391a13dc4aa436c44d8661b8b08ebef2326b51aeb46d220860f009c99288"
Oct 30 18:36:40.891756 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:40.891732    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jkgcq_openshift-console-operator(60e0e397-bb2c-4637-9376-c5af871666cb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq" podUID="60e0e397-bb2c-4637-9376-c5af871666cb"
Oct 30 18:36:41.706258 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:41.706229    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:41.706456 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:41.706263    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:41.706456 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:41.706291    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:41.706456 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:41.706337    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:41.706456 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:41.706369    4924 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:41.706456 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:41.706393    4924 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:41.706456 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:41.706410    4924 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b777d677-gcvwt: secret "image-registry-tls" not found
Oct 30 18:36:41.706456 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:41.706420    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls podName:12928ad7-4038-473a-a4aa-be1a6a42eabc nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.70640599 +0000 UTC m=+4.983456413 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls") pod "dns-default-sxk64" (UID: "12928ad7-4038-473a-a4aa-be1a6a42eabc") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:41.706456 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:41.706421    4924 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:41.706456 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:41.706373    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6pktt\" (UID: \"9b63e008-3071-4593-bba8-3bcc673b5dc2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6pktt"
Oct 30 18:36:41.706456 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:41.706457    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls podName:be409e3c-3420-4250-bfa7-3b01ecd69d9b nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.706440403 +0000 UTC m=+4.983490826 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls") pod "image-registry-7b777d677-gcvwt" (UID: "be409e3c-3420-4250-bfa7-3b01ecd69d9b") : secret "image-registry-tls" not found
Oct 30 18:36:41.706456 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:41.706427    4924 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:36:41.706881 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:41.706489    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls podName:a11d470f-d198-42ec-bee1-a900c4814473 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.706478622 +0000 UTC m=+4.983529049 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xcwnk" (UID: "a11d470f-d198-42ec-bee1-a900c4814473") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:41.706881 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:41.706510    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert podName:9b63e008-3071-4593-bba8-3bcc673b5dc2 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.706499579 +0000 UTC m=+4.983550004 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6pktt" (UID: "9b63e008-3071-4593-bba8-3bcc673b5dc2") : secret "networking-console-plugin-cert" not found
Oct 30 18:36:41.706881 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:41.706513    4924 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:36:41.706881 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:41.706559    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2xb8v\" (UID: \"b94a90ae-e06a-4e60-a48e-b567ce6e4edd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"
Oct 30 18:36:41.706881 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:41.706607    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs podName:05fe2f04-29f7-4999-87ef-b30035b34883 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.706591558 +0000 UTC m=+4.983641986 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs") pod "router-default-b44dd54dc-hqb7p" (UID: "05fe2f04-29f7-4999-87ef-b30035b34883") : secret "router-metrics-certs-default" not found
Oct 30 18:36:41.706881 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:41.706631    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:41.706881 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:41.706654    4924 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:36:41.706881 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:41.706687    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:41.706881 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:41.706714    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:41.706881 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:41.706730    4924 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:41.706881 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:41.706737    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls podName:b94a90ae-e06a-4e60-a48e-b567ce6e4edd nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.706723877 +0000 UTC m=+4.983774304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-2xb8v" (UID: "b94a90ae-e06a-4e60-a48e-b567ce6e4edd") : secret "samples-operator-tls" not found
Oct 30 18:36:41.706881 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:41.706776    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm podName:a11d470f-d198-42ec-bee1-a900c4814473 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.706762976 +0000 UTC m=+4.983813400 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xcwnk" (UID: "a11d470f-d198-42ec-bee1-a900c4814473") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:41.706881 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:41.706780    4924 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:41.706881 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:41.706789    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle podName:05fe2f04-29f7-4999-87ef-b30035b34883 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.706783502 +0000 UTC m=+4.983833925 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle") pod "router-default-b44dd54dc-hqb7p" (UID: "05fe2f04-29f7-4999-87ef-b30035b34883") : configmap references non-existent config key: service-ca.crt
Oct 30 18:36:41.706881 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:41.706771    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs\") pod \"network-metrics-daemon-sqkfq\" (UID: \"6df0f1e7-07b7-4231-bc70-ba81aef29856\") " pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:41.706881 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:41.706793    4924 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8598c87c5b-d4m59: secret "image-registry-tls" not found
Oct 30 18:36:41.706881 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:41.706818    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-plkxx\" (UID: \"80e45018-1682-47cd-9523-dbeaf35a39cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:41.706881 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:41.706831    4924 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:36:41.706881 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:41.706833    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls podName:92981ab7-a41b-482f-98c3-a328e5e8d6f0 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.706822684 +0000 UTC m=+4.983873107 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls") pod "image-registry-8598c87c5b-d4m59" (UID: "92981ab7-a41b-482f-98c3-a328e5e8d6f0") : secret "image-registry-tls" not found
Oct 30 18:36:41.706881 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:41.706864    4924 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:41.706881 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:41.706870    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs podName:6df0f1e7-07b7-4231-bc70-ba81aef29856 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.706863363 +0000 UTC m=+4.983913787 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs") pod "network-metrics-daemon-sqkfq" (UID: "6df0f1e7-07b7-4231-bc70-ba81aef29856") : secret "metrics-daemon-secret" not found
Oct 30 18:36:41.706881 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:41.706888    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls podName:80e45018-1682-47cd-9523-dbeaf35a39cd nodeName:}" failed. No retries permitted until 2026-10-30 18:36:43.706880802 +0000 UTC m=+4.983931225 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-plkxx" (UID: "80e45018-1682-47cd-9523-dbeaf35a39cd") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:41.896075 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:41.896033    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 18:36:42.242713 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:42.242680    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:42.243048 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:42.243033    4924 scope.go:122] "RemoveContainer" containerID="6018391a13dc4aa436c44d8661b8b08ebef2326b51aeb46d220860f009c99288"
Oct 30 18:36:42.243237 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:42.243220    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jkgcq_openshift-console-operator(60e0e397-bb2c-4637-9376-c5af871666cb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq" podUID="60e0e397-bb2c-4637-9376-c5af871666cb"
Oct 30 18:36:42.456840 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:42.456811    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:42.957131 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:42.957095    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:43.157749 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:43.157718    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:43.459229 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:43.459198    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:43.635499 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:43.635465    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:43.723712 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:43.723632    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:43.723712 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:43.723681    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:43.723712 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:43.723713    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs\") pod \"network-metrics-daemon-sqkfq\" (UID: \"6df0f1e7-07b7-4231-bc70-ba81aef29856\") " pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:43.723974 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:43.723739    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-plkxx\" (UID: \"80e45018-1682-47cd-9523-dbeaf35a39cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:43.723974 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:43.723753    4924 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:43.723974 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:43.723791    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:43.723974 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:43.723813    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm podName:a11d470f-d198-42ec-bee1-a900c4814473 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:47.723796038 +0000 UTC m=+9.000846467 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xcwnk" (UID: "a11d470f-d198-42ec-bee1-a900c4814473") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:43.723974 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:43.723849    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:43.723974 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:43.723861    4924 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:43.723974 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:43.723862    4924 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:43.723974 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:43.723882    4924 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8598c87c5b-d4m59: secret "image-registry-tls" not found
Oct 30 18:36:43.723974 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:43.723886    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:43.723974 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:43.723894    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls podName:12928ad7-4038-473a-a4aa-be1a6a42eabc nodeName:}" failed. No retries permitted until 2026-10-30 18:36:47.723883839 +0000 UTC m=+9.000934276 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls") pod "dns-default-sxk64" (UID: "12928ad7-4038-473a-a4aa-be1a6a42eabc") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:43.723974 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:43.723904    4924 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:43.723974 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:43.723922    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:43.723974 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:43.723936    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls podName:92981ab7-a41b-482f-98c3-a328e5e8d6f0 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:47.723919073 +0000 UTC m=+9.000969500 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls") pod "image-registry-8598c87c5b-d4m59" (UID: "92981ab7-a41b-482f-98c3-a328e5e8d6f0") : secret "image-registry-tls" not found
Oct 30 18:36:43.723974 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:43.723949    4924 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:43.723974 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:43.723961    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls podName:80e45018-1682-47cd-9523-dbeaf35a39cd nodeName:}" failed. No retries permitted until 2026-10-30 18:36:47.723945596 +0000 UTC m=+9.000996021 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-plkxx" (UID: "80e45018-1682-47cd-9523-dbeaf35a39cd") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:43.723974 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:43.723974    4924 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:36:43.723974 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:43.723982    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls podName:a11d470f-d198-42ec-bee1-a900c4814473 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:47.723971956 +0000 UTC m=+9.001022413 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xcwnk" (UID: "a11d470f-d198-42ec-bee1-a900c4814473") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:43.724613 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:43.723991    4924 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:43.724613 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:43.724005    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs podName:05fe2f04-29f7-4999-87ef-b30035b34883 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:47.723994972 +0000 UTC m=+9.001045426 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs") pod "router-default-b44dd54dc-hqb7p" (UID: "05fe2f04-29f7-4999-87ef-b30035b34883") : secret "router-metrics-certs-default" not found
Oct 30 18:36:43.724613 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:43.724012    4924 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b777d677-gcvwt: secret "image-registry-tls" not found
Oct 30 18:36:43.724613 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:43.724031    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6pktt\" (UID: \"9b63e008-3071-4593-bba8-3bcc673b5dc2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6pktt"
Oct 30 18:36:43.724613 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:43.724053    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2xb8v\" (UID: \"b94a90ae-e06a-4e60-a48e-b567ce6e4edd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"
Oct 30 18:36:43.724613 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:43.724091    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls podName:be409e3c-3420-4250-bfa7-3b01ecd69d9b nodeName:}" failed. No retries permitted until 2026-10-30 18:36:47.724057159 +0000 UTC m=+9.001107598 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls") pod "image-registry-7b777d677-gcvwt" (UID: "be409e3c-3420-4250-bfa7-3b01ecd69d9b") : secret "image-registry-tls" not found
Oct 30 18:36:43.724613 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:43.724135    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:43.724613 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:43.724164    4924 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:36:43.724613 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:43.724160    4924 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 18:36:43.724613 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:43.724209    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert podName:9b63e008-3071-4593-bba8-3bcc673b5dc2 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:47.724196525 +0000 UTC m=+9.001246953 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6pktt" (UID: "9b63e008-3071-4593-bba8-3bcc673b5dc2") : secret "networking-console-plugin-cert" not found
Oct 30 18:36:43.724613 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:43.724251    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle podName:05fe2f04-29f7-4999-87ef-b30035b34883 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:47.724239091 +0000 UTC m=+9.001289539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle") pod "router-default-b44dd54dc-hqb7p" (UID: "05fe2f04-29f7-4999-87ef-b30035b34883") : configmap references non-existent config key: service-ca.crt
Oct 30 18:36:43.726291 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:43.726274    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df0f1e7-07b7-4231-bc70-ba81aef29856-metrics-certs\") pod \"network-metrics-daemon-sqkfq\" (UID: \"6df0f1e7-07b7-4231-bc70-ba81aef29856\") " pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:43.726291 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:43.726285    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b94a90ae-e06a-4e60-a48e-b567ce6e4edd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-2xb8v\" (UID: \"b94a90ae-e06a-4e60-a48e-b567ce6e4edd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"
Oct 30 18:36:43.833072 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:43.833050    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"
Oct 30 18:36:43.838460 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:43.838447    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sqkfq"
Oct 30 18:36:44.206508 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:44.206481    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:44.295949 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:44.295919    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v"]
Oct 30 18:36:44.296397 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:44.296377    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sqkfq"]
Oct 30 18:36:44.298941 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:44.298914    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df0f1e7_07b7_4231_bc70_ba81aef29856.slice/crio-c11b54ab79240b1eaebc9697471c3bed25662b18e3df59c0163a71b0cc1fc5ca WatchSource:0}: Error finding container c11b54ab79240b1eaebc9697471c3bed25662b18e3df59c0163a71b0cc1fc5ca: Status 404 returned error can't find the container with id c11b54ab79240b1eaebc9697471c3bed25662b18e3df59c0163a71b0cc1fc5ca
Oct 30 18:36:44.301023 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:44.301009    4924 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:36:44.406937 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:44.406910    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:44.807466 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:44.807438    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:44.907866 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:44.907832    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sqkfq" event={"ID":"6df0f1e7-07b7-4231-bc70-ba81aef29856","Type":"ContainerStarted","Data":"c11b54ab79240b1eaebc9697471c3bed25662b18e3df59c0163a71b0cc1fc5ca"}
Oct 30 18:36:44.908980 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:44.908958    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v" event={"ID":"b94a90ae-e06a-4e60-a48e-b567ce6e4edd","Type":"ContainerStarted","Data":"f936d5a076400a6bca503cac9201b1ab536d149884ae55ea18e914119b5d677c"}
Oct 30 18:36:45.004713 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:45.004686    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:45.253480 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:45.253451    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cdf94_90df09a3-8706-4883-b711-be63e8e9f191/dns-node-resolver/0.log"
Oct 30 18:36:45.419383 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:45.419351    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:45.644206 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:45.644142    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:45.914805 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:45.914774    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sqkfq" event={"ID":"6df0f1e7-07b7-4231-bc70-ba81aef29856","Type":"ContainerStarted","Data":"19a011751fbf07b9d675d9aef68592c7b49d13338eb52852eb656eb89656edd9"}
Oct 30 18:36:46.144547 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:46.143428    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:46.202201 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:46.202121    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:46.536986 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:46.536950    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:46.653039 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:46.653016    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vtp2j_b9849d01-613f-439d-86de-b517ee4a3341/node-ca/0.log"
Oct 30 18:36:46.746254 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:46.746184    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:46.920322 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:46.920290    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sqkfq" event={"ID":"6df0f1e7-07b7-4231-bc70-ba81aef29856","Type":"ContainerStarted","Data":"9a4521631bd2998b077d126a6257a96aea86a6794ebdfae75bf83b5dc2b4bd40"}
Oct 30 18:36:46.922073 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:46.922041    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v" event={"ID":"b94a90ae-e06a-4e60-a48e-b567ce6e4edd","Type":"ContainerStarted","Data":"2a6ed1c7092fde50dafcfe203a3bb1b38ec7c39356ac81ec7e786fa282e6b781"}
Oct 30 18:36:46.922176 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:46.922090    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-2xb8v" event={"ID":"b94a90ae-e06a-4e60-a48e-b567ce6e4edd","Type":"ContainerStarted","Data":"15c06685f4c6d82f33983a273c35c1014e7519f863485320174c87c7f13d1c4e"}
Oct 30 18:36:46.934741 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:46.934700    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:47.117779 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:47.117746    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:47.254631 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:47.254604    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g6sxb_21b6f672-2dc5-4523-9dfa-6dfaa30a8bac/serve-healthcheck-canary/0.log"
Oct 30 18:36:47.419786 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:47.419707    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:47.654255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:47.654228    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-dfv27_9d148c95-78ae-43af-8942-460853970292/migrator/0.log"
Oct 30 18:36:47.760677 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:47.760601    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:47.760677 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:47.760638    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:47.760677 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:47.760666    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:47.760942 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:47.760684    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:47.760942 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:47.760739    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6pktt\" (UID: \"9b63e008-3071-4593-bba8-3bcc673b5dc2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6pktt"
Oct 30 18:36:47.760942 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:47.760802    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:47.761170 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:47.760942    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle podName:05fe2f04-29f7-4999-87ef-b30035b34883 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:55.760923342 +0000 UTC m=+17.037973781 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle") pod "router-default-b44dd54dc-hqb7p" (UID: "05fe2f04-29f7-4999-87ef-b30035b34883") : configmap references non-existent config key: service-ca.crt
Oct 30 18:36:47.761170 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:47.760946    4924 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:47.761170 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:47.761051    4924 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:47.761170 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:47.761033    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:47.761170 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:47.761126    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm podName:a11d470f-d198-42ec-bee1-a900c4814473 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:55.761105517 +0000 UTC m=+17.038155953 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xcwnk" (UID: "a11d470f-d198-42ec-bee1-a900c4814473") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:47.761170 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:47.761154    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls podName:a11d470f-d198-42ec-bee1-a900c4814473 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:55.761143784 +0000 UTC m=+17.038194208 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xcwnk" (UID: "a11d470f-d198-42ec-bee1-a900c4814473") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:47.761404 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:47.761188    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:47.761404 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:47.761224    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-plkxx\" (UID: \"80e45018-1682-47cd-9523-dbeaf35a39cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:47.761404 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:47.761341    4924 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:47.761404 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:47.761380    4924 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls podName:80e45018-1682-47cd-9523-dbeaf35a39cd nodeName:}" failed. No retries permitted until 2026-10-30 18:36:55.761367863 +0000 UTC m=+17.038418316 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-plkxx" (UID: "80e45018-1682-47cd-9523-dbeaf35a39cd") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:47.763277 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:47.763251    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fe2f04-29f7-4999-87ef-b30035b34883-metrics-certs\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:47.763383 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:47.763335    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls\") pod \"image-registry-7b777d677-gcvwt\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") " pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:47.763439 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:47.763420    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12928ad7-4038-473a-a4aa-be1a6a42eabc-metrics-tls\") pod \"dns-default-sxk64\" (UID: \"12928ad7-4038-473a-a4aa-be1a6a42eabc\") " pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:47.763889 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:47.763872    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b63e008-3071-4593-bba8-3bcc673b5dc2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6pktt\" (UID: \"9b63e008-3071-4593-bba8-3bcc673b5dc2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6pktt"
Oct 30 18:36:47.763962 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:47.763944    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls\") pod \"image-registry-8598c87c5b-d4m59\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") " pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:48.035616 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:48.035537    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6pktt"
Oct 30 18:36:48.035616 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:48.035574    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:48.045570 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:48.045542    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:48.045678 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:48.045627    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:48.054962 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:48.054943    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk_0648f559-00ac-4de0-8b9f-ce02d1b657a9/kube-storage-version-migrator-operator/0.log"
Oct 30 18:36:48.109892 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:48.109841    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:48.110090 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:48.110076    4924 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:36:48.212188 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:48.212154    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:48.406011 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:48.405980    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6pktt"]
Oct 30 18:36:48.408776 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:48.408749    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b63e008_3071_4593_bba8_3bcc673b5dc2.slice/crio-bf124b810f3aacc3274fe0c82fdf285392032575027a11e15b37c030b21ca480 WatchSource:0}: Error finding container bf124b810f3aacc3274fe0c82fdf285392032575027a11e15b37c030b21ca480: Status 404 returned error can't find the container with id bf124b810f3aacc3274fe0c82fdf285392032575027a11e15b37c030b21ca480
Oct 30 18:36:48.452861 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:48.452833    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sxk64"]
Oct 30 18:36:48.456198 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:48.456173    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12928ad7_4038_473a_a4aa_be1a6a42eabc.slice/crio-f09bccbf0178ee12e8433077d769ff7f84d8dedcd3ac0df78582105430b48e4d WatchSource:0}: Error finding container f09bccbf0178ee12e8433077d769ff7f84d8dedcd3ac0df78582105430b48e4d: Status 404 returned error can't find the container with id f09bccbf0178ee12e8433077d769ff7f84d8dedcd3ac0df78582105430b48e4d
Oct 30 18:36:48.456384 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:48.456351    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8598c87c5b-d4m59"]
Oct 30 18:36:48.460123 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:48.460099    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92981ab7_a41b_482f_98c3_a328e5e8d6f0.slice/crio-67404a616c085b88cb3827d686298ac30031ce0cd8da1a6cba75d8bd04775c45 WatchSource:0}: Error finding container 67404a616c085b88cb3827d686298ac30031ce0cd8da1a6cba75d8bd04775c45: Status 404 returned error can't find the container with id 67404a616c085b88cb3827d686298ac30031ce0cd8da1a6cba75d8bd04775c45
Oct 30 18:36:48.478479 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:48.478445    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b777d677-gcvwt"]
Oct 30 18:36:48.486097 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:48.486045    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe409e3c_3420_4250_bfa7_3b01ecd69d9b.slice/crio-5d12c88a7788bf4b0b5e289cfdc4ccac8f62cc32cebfaf0d8b3b1c64dc3bc586 WatchSource:0}: Error finding container 5d12c88a7788bf4b0b5e289cfdc4ccac8f62cc32cebfaf0d8b3b1c64dc3bc586: Status 404 returned error can't find the container with id 5d12c88a7788bf4b0b5e289cfdc4ccac8f62cc32cebfaf0d8b3b1c64dc3bc586
Oct 30 18:36:48.595571 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:48.595531    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:48.764545 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:48.764465    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:36:48.804952 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:48.804860    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:48.893657 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:48.893627    4924 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:48.893981 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:48.893968    4924 scope.go:122] "RemoveContainer" containerID="6018391a13dc4aa436c44d8661b8b08ebef2326b51aeb46d220860f009c99288"
Oct 30 18:36:48.894180 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:36:48.894161    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jkgcq_openshift-console-operator(60e0e397-bb2c-4637-9376-c5af871666cb)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq" podUID="60e0e397-bb2c-4637-9376-c5af871666cb"
Oct 30 18:36:48.929486 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:48.929452    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sxk64" event={"ID":"12928ad7-4038-473a-a4aa-be1a6a42eabc","Type":"ContainerStarted","Data":"f09bccbf0178ee12e8433077d769ff7f84d8dedcd3ac0df78582105430b48e4d"}
Oct 30 18:36:48.930683 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:48.930654    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-6pktt" event={"ID":"9b63e008-3071-4593-bba8-3bcc673b5dc2","Type":"ContainerStarted","Data":"bf124b810f3aacc3274fe0c82fdf285392032575027a11e15b37c030b21ca480"}
Oct 30 18:36:48.932286 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:48.932258    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8598c87c5b-d4m59" event={"ID":"92981ab7-a41b-482f-98c3-a328e5e8d6f0","Type":"ContainerStarted","Data":"b2ee92f7ee5759892c0041274a5f600243b873db3a8e49279354fc231827bbbf"}
Oct 30 18:36:48.932397 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:48.932301    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8598c87c5b-d4m59" event={"ID":"92981ab7-a41b-482f-98c3-a328e5e8d6f0","Type":"ContainerStarted","Data":"67404a616c085b88cb3827d686298ac30031ce0cd8da1a6cba75d8bd04775c45"}
Oct 30 18:36:48.933900 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:48.933874    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b777d677-gcvwt" event={"ID":"be409e3c-3420-4250-bfa7-3b01ecd69d9b","Type":"ContainerStarted","Data":"da9398581e049624eff438d8fc0366d75919657cea7ce3bbcd953f8e68c9c6d3"}
Oct 30 18:36:48.933991 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:48.933920    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b777d677-gcvwt" event={"ID":"be409e3c-3420-4250-bfa7-3b01ecd69d9b","Type":"ContainerStarted","Data":"5d12c88a7788bf4b0b5e289cfdc4ccac8f62cc32cebfaf0d8b3b1c64dc3bc586"}
Oct 30 18:36:48.960690 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:48.960659    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mgd4r"
Oct 30 18:36:49.004384 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:49.004353    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:49.195848 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:49.195817    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:49.937643 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:49.937610    4924 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:36:49.937643 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:49.937627    4924 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:36:50.375446 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:50.375407    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:50.559048 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:50.559023    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:50.737655 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:50.737618    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:50.827221 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:50.827192    4924 scope.go:122] "RemoveContainer" containerID="6018391a13dc4aa436c44d8661b8b08ebef2326b51aeb46d220860f009c99288"
Oct 30 18:36:50.920946 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:50.920864    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:50.942286 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:50.942255    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-6pktt" event={"ID":"9b63e008-3071-4593-bba8-3bcc673b5dc2","Type":"ContainerStarted","Data":"c619858d0e0eb1a3c3ec9994e95c84f159689f5da7e0f2eb09c088a4d6a0be9c"}
Oct 30 18:36:50.944233 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:50.944215    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 18:36:50.944351 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:50.944307    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq" event={"ID":"60e0e397-bb2c-4637-9376-c5af871666cb","Type":"ContainerStarted","Data":"1d4cf46b186ed2ea4d64a344e174e185201dd8f0bef31778f55edf4f63c0b614"}
Oct 30 18:36:50.944599 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:50.944575    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:50.945805 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:50.945784    4924 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-jkgcq container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused" start-of-body=
Oct 30 18:36:50.945910 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:50.945831    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq" podUID="60e0e397-bb2c-4637-9376-c5af871666cb" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused"
Oct 30 18:36:50.946273 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:50.946254    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sxk64" event={"ID":"12928ad7-4038-473a-a4aa-be1a6a42eabc","Type":"ContainerStarted","Data":"922d81bcba200721612ed783c7ea5d7e68db65abdb1e3f16e1f6d2f28338a7ab"}
Oct 30 18:36:50.946328 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:50.946289    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sxk64" event={"ID":"12928ad7-4038-473a-a4aa-be1a6a42eabc","Type":"ContainerStarted","Data":"24245345b697bd3bf3e9a8a3393895ae469e4af704e786be40782eaec395be5a"}
Oct 30 18:36:50.946409 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:50.946394    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sxk64"
Oct 30 18:36:51.109999 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:51.109969    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:51.288962 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:51.288880    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:51.717830 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:51.717801    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:51.905904 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:51.905871    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:36:51.953455 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:51.953427    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jkgcq"
Oct 30 18:36:52.405901 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:52.405869    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:52.575883 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:52.575851    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:52.685756 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:52.685686    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:36:52.757105 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:52.757052    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:52.939429 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:52.939350    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:53.134077 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:53.134020    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:53.325413 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:53.325332    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:53.509581 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:53.509547    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:53.694471 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:53.694445    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:53.963329 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:53.963256    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5v2r9"
Oct 30 18:36:54.116528 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:54.116496    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:54.299291 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:54.299193    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:54.563315 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:54.563217    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:54.743362 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:54.743325    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:55.297986 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:55.297949    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8598c87c5b-d4m59"]
Oct 30 18:36:55.523169 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:55.523125    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:55.708692 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:55.708664    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:55.832025 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:55.831991    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:55.832198 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:55.832038    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:55.832198 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:55.832095    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:55.832198 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:55.832120    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-plkxx\" (UID: \"80e45018-1682-47cd-9523-dbeaf35a39cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:55.832638 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:55.832613    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05fe2f04-29f7-4999-87ef-b30035b34883-service-ca-bundle\") pod \"router-default-b44dd54dc-hqb7p\" (UID: \"05fe2f04-29f7-4999-87ef-b30035b34883\") " pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:55.832638 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:55.832636    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a11d470f-d198-42ec-bee1-a900c4814473-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:55.834435 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:55.834415    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/80e45018-1682-47cd-9523-dbeaf35a39cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-plkxx\" (UID: \"80e45018-1682-47cd-9523-dbeaf35a39cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:55.834521 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:55.834492    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a11d470f-d198-42ec-bee1-a900c4814473-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xcwnk\" (UID: \"a11d470f-d198-42ec-bee1-a900c4814473\") " pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:55.847330 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:55.847309    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xcwnk"
Oct 30 18:36:55.993257 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:55.993182    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:56.133092 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:56.133043    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:56.133441 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:56.133118    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"
Oct 30 18:36:56.189789 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:56.189678    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:56.212169 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:56.212136    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xcwnk"]
Oct 30 18:36:56.245762 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:56.245736    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11d470f_d198_42ec_bee1_a900c4814473.slice/crio-dff29ad48cab1cfe0ae7184c349692893a611010d5f49869404f87c7196c2558 WatchSource:0}: Error finding container dff29ad48cab1cfe0ae7184c349692893a611010d5f49869404f87c7196c2558: Status 404 returned error can't find the container with id dff29ad48cab1cfe0ae7184c349692893a611010d5f49869404f87c7196c2558
Oct 30 18:36:56.503772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:56.503723    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-b44dd54dc-hqb7p"]
Oct 30 18:36:56.506296 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:56.506267    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05fe2f04_29f7_4999_87ef_b30035b34883.slice/crio-88d0537d5bd88cae09f7ef537e3aa7e22e691f8d2962523a05437aafc67d7a8d WatchSource:0}: Error finding container 88d0537d5bd88cae09f7ef537e3aa7e22e691f8d2962523a05437aafc67d7a8d: Status 404 returned error can't find the container with id 88d0537d5bd88cae09f7ef537e3aa7e22e691f8d2962523a05437aafc67d7a8d
Oct 30 18:36:56.546555 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:56.546532    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx"]
Oct 30 18:36:56.548931 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:36:56.548900    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e45018_1682_47cd_9523_dbeaf35a39cd.slice/crio-b501fd4c510386ec5e3f53b4d4987806dc61a46796e4d9d1024d9aa74821a6f8 WatchSource:0}: Error finding container b501fd4c510386ec5e3f53b4d4987806dc61a46796e4d9d1024d9aa74821a6f8: Status 404 returned error can't find the container with id b501fd4c510386ec5e3f53b4d4987806dc61a46796e4d9d1024d9aa74821a6f8
Oct 30 18:36:56.967779 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:56.967742    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xcwnk" event={"ID":"a11d470f-d198-42ec-bee1-a900c4814473","Type":"ContainerStarted","Data":"4cecfc4af8aca6d4e500bc6780116d59055dc426cf217f08937cbcf136a7791b"}
Oct 30 18:36:56.967968 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:56.967796    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xcwnk" event={"ID":"a11d470f-d198-42ec-bee1-a900c4814473","Type":"ContainerStarted","Data":"dff29ad48cab1cfe0ae7184c349692893a611010d5f49869404f87c7196c2558"}
Oct 30 18:36:56.969236 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:56.969206    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-b44dd54dc-hqb7p" event={"ID":"05fe2f04-29f7-4999-87ef-b30035b34883","Type":"ContainerStarted","Data":"0901b27a78f8cd50cf37e6f2d0f24c617c5df73ecae2b8b99647ee7e6d67ec95"}
Oct 30 18:36:56.969360 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:56.969253    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-b44dd54dc-hqb7p" event={"ID":"05fe2f04-29f7-4999-87ef-b30035b34883","Type":"ContainerStarted","Data":"88d0537d5bd88cae09f7ef537e3aa7e22e691f8d2962523a05437aafc67d7a8d"}
Oct 30 18:36:56.970301 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:56.970275    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx" event={"ID":"80e45018-1682-47cd-9523-dbeaf35a39cd","Type":"ContainerStarted","Data":"b501fd4c510386ec5e3f53b4d4987806dc61a46796e4d9d1024d9aa74821a6f8"}
Oct 30 18:36:57.124081 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:57.124044    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:57.134023 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:57.134000    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:57.136392 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:57.136371    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:57.306678 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:57.306470    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:57.976818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:57.976780    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xcwnk" event={"ID":"a11d470f-d198-42ec-bee1-a900c4814473","Type":"ContainerStarted","Data":"7648912716fccc1925f5949945b8eb3d25386e81ecdd5319f3ac648c7cb86ed7"}
Oct 30 18:36:57.977211 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:57.976994    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:57.978172 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:57.978152    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-b44dd54dc-hqb7p"
Oct 30 18:36:59.989767 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:59.989730    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-plkxx" event={"ID":"80e45018-1682-47cd-9523-dbeaf35a39cd","Type":"ContainerStarted","Data":"c7b0eaca975276ddb60858aa93670267cb5ddb78116745f8b07e67f34dfa5036"}
Oct 30 18:36:59.991797 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:36:59.991773    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xcwnk" event={"ID":"a11d470f-d198-42ec-bee1-a900c4814473","Type":"ContainerStarted","Data":"19f31bf1ed0cc61bf3e475ae2d5233ba2333499eadb2be93997eff56ad0cdc81"}
Oct 30 18:37:00.426684 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:00.426641    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:00.692819 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:00.692713    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:00.877470 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:00.877404    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:00.951753 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:00.951674    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sxk64"
Oct 30 18:37:01.535678 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:01.535635    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:01.724823 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:01.724783    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:02.922170 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:02.922129    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:03.104886 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:03.104856    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:04.818622 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:04.818552    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:05.001274 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:05.001244    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:05.194484 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:05.194429    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:05.303522 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:05.303497    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:37:05.385932 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:05.385898    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:07.389468 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:07.389428    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:07.567994 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:07.567941    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:08.767880 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:08.767849    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xcmps"
Oct 30 18:37:10.146283 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:10.146254    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:10.631365 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:10.631321    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:10.832248 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:10.832218    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:11.039455 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:11.039369    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:11.237936 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:11.237886    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:11.424633 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:11.424590    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:11.628883 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:11.628830    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:11.812458 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:11.812382    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:11.869634 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:11.869603    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jrwzp"]
Oct 30 18:37:11.902178 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:11.902153    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:11.910654 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:11.910633    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:37:11.961876 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:11.961845    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/93234f2e-e57d-4698-8206-a4d00a8095f1-node-exporter-textfile\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:11.962001 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:11.961890    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/93234f2e-e57d-4698-8206-a4d00a8095f1-root\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:11.962052 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:11.961996    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcgfm\" (UniqueName: \"kubernetes.io/projected/93234f2e-e57d-4698-8206-a4d00a8095f1-kube-api-access-zcgfm\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:11.962114 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:11.962048    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93234f2e-e57d-4698-8206-a4d00a8095f1-metrics-client-ca\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:11.962183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:11.962161    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93234f2e-e57d-4698-8206-a4d00a8095f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:11.962227 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:11.962210    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/93234f2e-e57d-4698-8206-a4d00a8095f1-node-exporter-wtmp\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:11.962269 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:11.962249    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/93234f2e-e57d-4698-8206-a4d00a8095f1-node-exporter-tls\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:11.962497 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:11.962463    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/93234f2e-e57d-4698-8206-a4d00a8095f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:11.962577 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:11.962547    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93234f2e-e57d-4698-8206-a4d00a8095f1-sys\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:12.002057 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.002029    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:12.009222 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.009204    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 18:37:12.009722 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.009704    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 18:37:12.055866 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.055842    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 18:37:12.056006 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.055896    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zw6ts\""
Oct 30 18:37:12.056006 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.055949    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 18:37:12.063752 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.063702    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/93234f2e-e57d-4698-8206-a4d00a8095f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:12.063752 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.063731    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93234f2e-e57d-4698-8206-a4d00a8095f1-sys\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:12.063955 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.063761    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/93234f2e-e57d-4698-8206-a4d00a8095f1-node-exporter-textfile\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:12.063955 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.063782    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/93234f2e-e57d-4698-8206-a4d00a8095f1-root\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:12.063955 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.063831    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93234f2e-e57d-4698-8206-a4d00a8095f1-sys\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:12.063955 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.063835    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/93234f2e-e57d-4698-8206-a4d00a8095f1-root\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:12.063955 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.063854    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcgfm\" (UniqueName: \"kubernetes.io/projected/93234f2e-e57d-4698-8206-a4d00a8095f1-kube-api-access-zcgfm\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:12.063955 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.063881    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93234f2e-e57d-4698-8206-a4d00a8095f1-metrics-client-ca\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:12.063955 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.063910    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93234f2e-e57d-4698-8206-a4d00a8095f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:12.063955 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.063938    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/93234f2e-e57d-4698-8206-a4d00a8095f1-node-exporter-wtmp\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:12.064391 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.063963    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/93234f2e-e57d-4698-8206-a4d00a8095f1-node-exporter-tls\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:12.064391 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.064202    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/93234f2e-e57d-4698-8206-a4d00a8095f1-node-exporter-wtmp\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:12.064497 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.064415    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/93234f2e-e57d-4698-8206-a4d00a8095f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:12.064636 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.064614    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/93234f2e-e57d-4698-8206-a4d00a8095f1-node-exporter-textfile\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:12.064830 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.064811    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93234f2e-e57d-4698-8206-a4d00a8095f1-metrics-client-ca\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:12.066249 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.066227    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93234f2e-e57d-4698-8206-a4d00a8095f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:12.066249 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.066241    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/93234f2e-e57d-4698-8206-a4d00a8095f1-node-exporter-tls\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:12.119438 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.119417    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcgfm\" (UniqueName: \"kubernetes.io/projected/93234f2e-e57d-4698-8206-a4d00a8095f1-kube-api-access-zcgfm\") pod \"node-exporter-jrwzp\" (UID: \"93234f2e-e57d-4698-8206-a4d00a8095f1\") " pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:12.192082 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.192028    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:12.254631 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.254601    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jrwzp"
Oct 30 18:37:12.294790 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:37:12.294760    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93234f2e_e57d_4698_8206_a4d00a8095f1.slice/crio-b9c8e028d35230682c07ba1198200208e722793b5621fd7f1674f330854005fe WatchSource:0}: Error finding container b9c8e028d35230682c07ba1198200208e722793b5621fd7f1674f330854005fe: Status 404 returned error can't find the container with id b9c8e028d35230682c07ba1198200208e722793b5621fd7f1674f330854005fe
Oct 30 18:37:12.383355 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.383317    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:12.690053 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.689967    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:12.878166 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:12.878128    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:13.040568 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:13.040492    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jrwzp" event={"ID":"93234f2e-e57d-4698-8206-a4d00a8095f1","Type":"ContainerStarted","Data":"b9c8e028d35230682c07ba1198200208e722793b5621fd7f1674f330854005fe"}
Oct 30 18:37:13.352527 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:13.352495    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:13.839462 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:13.839387    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:14.239895 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:14.239861    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:15.049412 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:15.049381    4924 generic.go:356] "Generic (PLEG): container finished" podID="93234f2e-e57d-4698-8206-a4d00a8095f1" containerID="70ffb430c2fe926fdf21228f42a88b9d6b9862c6127729cc6adb3869962ac555" exitCode=0
Oct 30 18:37:15.049854 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:15.049478    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jrwzp" event={"ID":"93234f2e-e57d-4698-8206-a4d00a8095f1","Type":"ContainerDied","Data":"70ffb430c2fe926fdf21228f42a88b9d6b9862c6127729cc6adb3869962ac555"}
Oct 30 18:37:15.710753 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:37:15.710713    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:37:16.056085 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:16.055990    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jrwzp" event={"ID":"93234f2e-e57d-4698-8206-a4d00a8095f1","Type":"ContainerStarted","Data":"6372ee8d844cc89e35d4789d87faaf3fcb7dd42c751bf29b35e5aa9aa198941a"}
Oct 30 18:37:16.056085 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:16.056051    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jrwzp" event={"ID":"93234f2e-e57d-4698-8206-a4d00a8095f1","Type":"ContainerStarted","Data":"a45c9cfa321d73f187471f8ee1209c9ff501fbac16e3cb3a1b6ff8330dda4d41"}
Oct 30 18:37:16.214695 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:16.214635    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jrwzp" podStartSLOduration=3.265003821 podStartE2EDuration="5.214617306s" totalImagesPullingTime="1.869667939s" totalInitContainerRuntime="79.945546ms" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:16.213957056 +0000 UTC m=+37.491007498" watchObservedRunningTime="2026-10-30 18:37:16.214617306 +0000 UTC m=+37.491667748"
Oct 30 18:37:16.222311 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:16.222278    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:16.425260 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:16.425203    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:16.617341 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:16.617294    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:16.918470 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:16.918442    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:17.118452 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:17.118407    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:17.895886 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:17.895855    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b777d677-gcvwt"]
Oct 30 18:37:18.048140 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:18.048100    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:18.248599 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:18.248515    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:18.726458 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:18.726431    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:18.908229 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:18.908199    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:20.317164 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.317060    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-8598c87c5b-d4m59" podUID="92981ab7-a41b-482f-98c3-a328e5e8d6f0" containerName="registry" containerID="cri-o://b2ee92f7ee5759892c0041274a5f600243b873db3a8e49279354fc231827bbbf" gracePeriod=30
Oct 30 18:37:20.590498 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.590441    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:37:20.736055 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.736014    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:20.749111 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.749083    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls\") pod \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") "
Oct 30 18:37:20.749252 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.749133    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92981ab7-a41b-482f-98c3-a328e5e8d6f0-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92981ab7-a41b-482f-98c3-a328e5e8d6f0-trusted-ca\") pod \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") "
Oct 30 18:37:20.749252 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.749155    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-bound-sa-token\") pod \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") "
Oct 30 18:37:20.749252 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.749192    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92981ab7-a41b-482f-98c3-a328e5e8d6f0-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/92981ab7-a41b-482f-98c3-a328e5e8d6f0-installation-pull-secrets\") pod \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") "
Oct 30 18:37:20.749252 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.749210    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92981ab7-a41b-482f-98c3-a328e5e8d6f0-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/92981ab7-a41b-482f-98c3-a328e5e8d6f0-image-registry-private-configuration\") pod \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") "
Oct 30 18:37:20.749430 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.749249    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-certificates\") pod \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") "
Oct 30 18:37:20.749430 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.749284    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-kube-api-access-dth6b\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-kube-api-access-dth6b\") pod \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") "
Oct 30 18:37:20.749430 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.749332    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92981ab7-a41b-482f-98c3-a328e5e8d6f0-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/92981ab7-a41b-482f-98c3-a328e5e8d6f0-ca-trust-extracted\") pod \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\" (UID: \"92981ab7-a41b-482f-98c3-a328e5e8d6f0\") "
Oct 30 18:37:20.750165 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.750040    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-certificates" pod "92981ab7-a41b-482f-98c3-a328e5e8d6f0" (UID: "92981ab7-a41b-482f-98c3-a328e5e8d6f0"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:37:20.750165 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.750047    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92981ab7-a41b-482f-98c3-a328e5e8d6f0-trusted-ca" pod "92981ab7-a41b-482f-98c3-a328e5e8d6f0" (UID: "92981ab7-a41b-482f-98c3-a328e5e8d6f0"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:37:20.752561 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.752532    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls" pod "92981ab7-a41b-482f-98c3-a328e5e8d6f0" (UID: "92981ab7-a41b-482f-98c3-a328e5e8d6f0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:37:20.756641 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.752857    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-bound-sa-token" pod "92981ab7-a41b-482f-98c3-a328e5e8d6f0" (UID: "92981ab7-a41b-482f-98c3-a328e5e8d6f0"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:37:20.756743 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.752887    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92981ab7-a41b-482f-98c3-a328e5e8d6f0-installation-pull-secrets" pod "92981ab7-a41b-482f-98c3-a328e5e8d6f0" (UID: "92981ab7-a41b-482f-98c3-a328e5e8d6f0"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:37:20.757258 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.757231    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-kube-api-access-dth6b" pod "92981ab7-a41b-482f-98c3-a328e5e8d6f0" (UID: "92981ab7-a41b-482f-98c3-a328e5e8d6f0"). InnerVolumeSpecName "kube-api-access-dth6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:37:20.757911 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.757893    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92981ab7-a41b-482f-98c3-a328e5e8d6f0-image-registry-private-configuration" pod "92981ab7-a41b-482f-98c3-a328e5e8d6f0" (UID: "92981ab7-a41b-482f-98c3-a328e5e8d6f0"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:37:20.758749 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.758725    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92981ab7-a41b-482f-98c3-a328e5e8d6f0-ca-trust-extracted" pod "92981ab7-a41b-482f-98c3-a328e5e8d6f0" (UID: "92981ab7-a41b-482f-98c3-a328e5e8d6f0"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:37:20.851026 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.850996    4924 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-tls\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:20.851026 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.851024    4924 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/92981ab7-a41b-482f-98c3-a328e5e8d6f0-trusted-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:20.851224 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.851037    4924 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-bound-sa-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:20.851224 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.851051    4924 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/92981ab7-a41b-482f-98c3-a328e5e8d6f0-installation-pull-secrets\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:20.851224 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.851090    4924 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/92981ab7-a41b-482f-98c3-a328e5e8d6f0-image-registry-private-configuration\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:20.851224 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.851105    4924 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/92981ab7-a41b-482f-98c3-a328e5e8d6f0-registry-certificates\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:20.851224 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.851118    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dth6b\" (UniqueName: \"kubernetes.io/projected/92981ab7-a41b-482f-98c3-a328e5e8d6f0-kube-api-access-dth6b\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:20.851224 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.851130    4924 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/92981ab7-a41b-482f-98c3-a328e5e8d6f0-ca-trust-extracted\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:20.938027 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:20.937990    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:21.079194 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:21.079162    4924 generic.go:356] "Generic (PLEG): container finished" podID="92981ab7-a41b-482f-98c3-a328e5e8d6f0" containerID="b2ee92f7ee5759892c0041274a5f600243b873db3a8e49279354fc231827bbbf" exitCode=0
Oct 30 18:37:21.079377 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:21.079234    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8598c87c5b-d4m59"
Oct 30 18:37:21.079377 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:21.079255    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8598c87c5b-d4m59" event={"ID":"92981ab7-a41b-482f-98c3-a328e5e8d6f0","Type":"ContainerDied","Data":"b2ee92f7ee5759892c0041274a5f600243b873db3a8e49279354fc231827bbbf"}
Oct 30 18:37:21.079377 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:21.079311    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8598c87c5b-d4m59" event={"ID":"92981ab7-a41b-482f-98c3-a328e5e8d6f0","Type":"ContainerDied","Data":"67404a616c085b88cb3827d686298ac30031ce0cd8da1a6cba75d8bd04775c45"}
Oct 30 18:37:21.079377 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:21.079332    4924 scope.go:122] "RemoveContainer" containerID="b2ee92f7ee5759892c0041274a5f600243b873db3a8e49279354fc231827bbbf"
Oct 30 18:37:21.087313 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:21.087278    4924 scope.go:122] "RemoveContainer" containerID="b2ee92f7ee5759892c0041274a5f600243b873db3a8e49279354fc231827bbbf"
Oct 30 18:37:21.087662 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:37:21.087635    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2ee92f7ee5759892c0041274a5f600243b873db3a8e49279354fc231827bbbf\": container with ID starting with b2ee92f7ee5759892c0041274a5f600243b873db3a8e49279354fc231827bbbf not found: ID does not exist" containerID="b2ee92f7ee5759892c0041274a5f600243b873db3a8e49279354fc231827bbbf"
Oct 30 18:37:21.087762 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:21.087675    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2ee92f7ee5759892c0041274a5f600243b873db3a8e49279354fc231827bbbf"} err="failed to get container status \"b2ee92f7ee5759892c0041274a5f600243b873db3a8e49279354fc231827bbbf\": rpc error: code = NotFound desc = could not find container \"b2ee92f7ee5759892c0041274a5f600243b873db3a8e49279354fc231827bbbf\": container with ID starting with b2ee92f7ee5759892c0041274a5f600243b873db3a8e49279354fc231827bbbf not found: ID does not exist"
Oct 30 18:37:21.136678 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:21.136606    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:21.238316 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:21.238284    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8598c87c5b-d4m59"]
Oct 30 18:37:21.284799 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:21.284769    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8598c87c5b-d4m59"]
Oct 30 18:37:22.832113 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:22.832084    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92981ab7-a41b-482f-98c3-a328e5e8d6f0" path="/var/lib/kubelet/pods/92981ab7-a41b-482f-98c3-a328e5e8d6f0/volumes"
Oct 30 18:37:27.252907 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:27.252869    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:27.438265 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:27.438222    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:28.330368 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:28.330328    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:28.511036 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:28.511001    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:28.715032 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:28.714998    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:28.913926 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:28.913890    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:29.107660 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:29.107626    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:29.588185 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:29.588089    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:29.768817 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:29.768763    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:32.496809 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:32.496776    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:32.688257 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:32.688223    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:37.397207 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:37.397173    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:37.586548 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:37.586505    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:42.918336 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:42.918280    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7b777d677-gcvwt" podUID="be409e3c-3420-4250-bfa7-3b01ecd69d9b" containerName="registry" containerID="cri-o://da9398581e049624eff438d8fc0366d75919657cea7ce3bbcd953f8e68c9c6d3" gracePeriod=30
Oct 30 18:37:43.162507 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.162479    4924 generic.go:356] "Generic (PLEG): container finished" podID="be409e3c-3420-4250-bfa7-3b01ecd69d9b" containerID="da9398581e049624eff438d8fc0366d75919657cea7ce3bbcd953f8e68c9c6d3" exitCode=0
Oct 30 18:37:43.162665 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.162555    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b777d677-gcvwt" event={"ID":"be409e3c-3420-4250-bfa7-3b01ecd69d9b","Type":"ContainerDied","Data":"da9398581e049624eff438d8fc0366d75919657cea7ce3bbcd953f8e68c9c6d3"}
Oct 30 18:37:43.162665 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.162615    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b777d677-gcvwt" event={"ID":"be409e3c-3420-4250-bfa7-3b01ecd69d9b","Type":"ContainerDied","Data":"5d12c88a7788bf4b0b5e289cfdc4ccac8f62cc32cebfaf0d8b3b1c64dc3bc586"}
Oct 30 18:37:43.162665 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.162629    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d12c88a7788bf4b0b5e289cfdc4ccac8f62cc32cebfaf0d8b3b1c64dc3bc586"
Oct 30 18:37:43.171567 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.171517    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:37:43.219429 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.219404    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be409e3c-3420-4250-bfa7-3b01ecd69d9b-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be409e3c-3420-4250-bfa7-3b01ecd69d9b-image-registry-private-configuration\") pod \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") "
Oct 30 18:37:43.219576 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.219449    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be409e3c-3420-4250-bfa7-3b01ecd69d9b-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be409e3c-3420-4250-bfa7-3b01ecd69d9b-trusted-ca\") pod \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") "
Oct 30 18:37:43.219576 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.219478    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be409e3c-3420-4250-bfa7-3b01ecd69d9b-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be409e3c-3420-4250-bfa7-3b01ecd69d9b-ca-trust-extracted\") pod \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") "
Oct 30 18:37:43.219576 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.219500    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-certificates\") pod \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") "
Oct 30 18:37:43.219576 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.219526    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be409e3c-3420-4250-bfa7-3b01ecd69d9b-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be409e3c-3420-4250-bfa7-3b01ecd69d9b-installation-pull-secrets\") pod \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") "
Oct 30 18:37:43.219576 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.219566    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls\") pod \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") "
Oct 30 18:37:43.219880 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.219592    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-kube-api-access-knktl\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-kube-api-access-knktl\") pod \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") "
Oct 30 18:37:43.219880 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.219619    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-bound-sa-token\") pod \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\" (UID: \"be409e3c-3420-4250-bfa7-3b01ecd69d9b\") "
Oct 30 18:37:43.220096 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.220040    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-certificates" pod "be409e3c-3420-4250-bfa7-3b01ecd69d9b" (UID: "be409e3c-3420-4250-bfa7-3b01ecd69d9b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:37:43.220212 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.220125    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be409e3c-3420-4250-bfa7-3b01ecd69d9b-trusted-ca" pod "be409e3c-3420-4250-bfa7-3b01ecd69d9b" (UID: "be409e3c-3420-4250-bfa7-3b01ecd69d9b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:37:43.222155 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.222123    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls" pod "be409e3c-3420-4250-bfa7-3b01ecd69d9b" (UID: "be409e3c-3420-4250-bfa7-3b01ecd69d9b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:37:43.222272 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.222248    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be409e3c-3420-4250-bfa7-3b01ecd69d9b-installation-pull-secrets" pod "be409e3c-3420-4250-bfa7-3b01ecd69d9b" (UID: "be409e3c-3420-4250-bfa7-3b01ecd69d9b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:37:43.222369 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.222348    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-kube-api-access-knktl" pod "be409e3c-3420-4250-bfa7-3b01ecd69d9b" (UID: "be409e3c-3420-4250-bfa7-3b01ecd69d9b"). InnerVolumeSpecName "kube-api-access-knktl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:37:43.222420 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.222348    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be409e3c-3420-4250-bfa7-3b01ecd69d9b-image-registry-private-configuration" pod "be409e3c-3420-4250-bfa7-3b01ecd69d9b" (UID: "be409e3c-3420-4250-bfa7-3b01ecd69d9b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:37:43.222420 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.222388    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-bound-sa-token" pod "be409e3c-3420-4250-bfa7-3b01ecd69d9b" (UID: "be409e3c-3420-4250-bfa7-3b01ecd69d9b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:37:43.226393 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.226374    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be409e3c-3420-4250-bfa7-3b01ecd69d9b-ca-trust-extracted" pod "be409e3c-3420-4250-bfa7-3b01ecd69d9b" (UID: "be409e3c-3420-4250-bfa7-3b01ecd69d9b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:37:43.320450 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.320411    4924 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be409e3c-3420-4250-bfa7-3b01ecd69d9b-ca-trust-extracted\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:43.320450 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.320445    4924 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-certificates\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:43.320450 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.320455    4924 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be409e3c-3420-4250-bfa7-3b01ecd69d9b-installation-pull-secrets\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:43.320649 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.320465    4924 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-registry-tls\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:43.320649 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.320475    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knktl\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-kube-api-access-knktl\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:43.320649 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.320483    4924 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be409e3c-3420-4250-bfa7-3b01ecd69d9b-bound-sa-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:43.320649 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.320493    4924 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be409e3c-3420-4250-bfa7-3b01ecd69d9b-image-registry-private-configuration\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:43.320649 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:43.320502    4924 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be409e3c-3420-4250-bfa7-3b01ecd69d9b-trusted-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:44.165651 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:44.165621    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b777d677-gcvwt"
Oct 30 18:37:44.368330 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:44.368294    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b777d677-gcvwt"]
Oct 30 18:37:44.414832 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:44.414802    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7b777d677-gcvwt"]
Oct 30 18:37:44.831583 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:44.831546    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be409e3c-3420-4250-bfa7-3b01ecd69d9b" path="/var/lib/kubelet/pods/be409e3c-3420-4250-bfa7-3b01ecd69d9b/volumes"
Oct 30 18:37:45.709465 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:37:45.709430    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:37:48.181090 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:48.181043    4924 generic.go:356] "Generic (PLEG): container finished" podID="0648f559-00ac-4de0-8b9f-ce02d1b657a9" containerID="48fc696af2d22b8f8e4aba80ab36337ca1f67a84bd21a7b6810c28457b1f3ed6" exitCode=0
Oct 30 18:37:48.181463 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:48.181117    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk" event={"ID":"0648f559-00ac-4de0-8b9f-ce02d1b657a9","Type":"ContainerDied","Data":"48fc696af2d22b8f8e4aba80ab36337ca1f67a84bd21a7b6810c28457b1f3ed6"}
Oct 30 18:37:48.181463 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:48.181450    4924 scope.go:122] "RemoveContainer" containerID="48fc696af2d22b8f8e4aba80ab36337ca1f67a84bd21a7b6810c28457b1f3ed6"
Oct 30 18:37:49.185809 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:49.185772    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-j6kdk" event={"ID":"0648f559-00ac-4de0-8b9f-ce02d1b657a9","Type":"ContainerStarted","Data":"df984dc0d7848441ff16d8ea9cb16f7bebfa6f4249c1f8f4d359612addc8fc87"}
Oct 30 18:37:50.968743 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:50.968704    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:51.147181 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:51.147149    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:52.399056 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:52.399023    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:52.576087 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:52.576025    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:55.206879 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:55.206845    4924 generic.go:356] "Generic (PLEG): container finished" podID="e954d56b-ace6-4539-a467-486c1e889886" containerID="9db3b68d4c5b11edd90b0d14940978f3ee6d2e0105105b15cc9a9c7520f86a1f" exitCode=0
Oct 30 18:37:55.207252 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:55.206919    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wpnq5" event={"ID":"e954d56b-ace6-4539-a467-486c1e889886","Type":"ContainerDied","Data":"9db3b68d4c5b11edd90b0d14940978f3ee6d2e0105105b15cc9a9c7520f86a1f"}
Oct 30 18:37:55.207252 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:55.207251    4924 scope.go:122] "RemoveContainer" containerID="9db3b68d4c5b11edd90b0d14940978f3ee6d2e0105105b15cc9a9c7520f86a1f"
Oct 30 18:37:56.212672 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:37:56.212643    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wpnq5" event={"ID":"e954d56b-ace6-4539-a467-486c1e889886","Type":"ContainerStarted","Data":"a83a6364e24d5fff2e4db7ebbcfa48bd555f71c5268320276a18b2b03e24403d"}
Oct 30 18:38:00.915397 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:38:00.915361    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:01.102495 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:38:01.102463    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:02.033505 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:38:02.033473    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:02.221303 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:38:02.221269    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:02.445708 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:38:02.445676    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:02.629210 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:38:02.629175    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:05.248134 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:38:05.248103    4924 generic.go:356] "Generic (PLEG): container finished" podID="abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9" containerID="9bbb3a662c60e111c0291040c70cbbeee7efa3b3a15e3f0219c5db1145a2c0b6" exitCode=0
Oct 30 18:38:05.248547 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:38:05.248175    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-776vb" event={"ID":"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9","Type":"ContainerDied","Data":"9bbb3a662c60e111c0291040c70cbbeee7efa3b3a15e3f0219c5db1145a2c0b6"}
Oct 30 18:38:05.248547 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:38:05.248466    4924 scope.go:122] "RemoveContainer" containerID="9bbb3a662c60e111c0291040c70cbbeee7efa3b3a15e3f0219c5db1145a2c0b6"
Oct 30 18:38:06.254241 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:38:06.254200    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-776vb" event={"ID":"abb5e9b3-82ee-4bbe-bc5f-8c763d6120e9","Type":"ContainerStarted","Data":"fcf0b7cae9d5fc80e5bc7c7c9316496e9888380de70acb0045caea4e5c36e14d"}
Oct 30 18:38:13.527320 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:38:13.527284    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:13.748210 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:38:13.748172    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:15.710235 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:38:15.710198    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:38:22.417425 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:38:22.417378    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:22.603528 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:38:22.603487    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:27.779977 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:38:27.779944    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:28.869796 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:38:28.869760    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:38:45.709842 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:38:45.709805    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:39:15.709738 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:39:15.709698    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:39:25.946219 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:39:25.946185    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:39:29.219046 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:39:29.219011    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:39:45.709742 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:39:45.709701    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:40:01.048011 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:01.047977    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:40:01.257236 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:01.257203    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:40:15.712780 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:40:15.712739    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:40:27.938208 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:27.938112    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:40:29.316322 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:29.316275    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:40:29.563228 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:29.563195    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:40:31.161901 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:31.161868    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:40:31.361932 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:31.361902    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:40:35.608430 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:35.608394    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:40:35.857877 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:35.857842    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:40:36.373744 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:36.373708    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:40:36.567195 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:36.567160    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:40:37.782443 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:37.782414    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:40:38.425919 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:38.425888    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:40:43.314795 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:43.314758    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-zmwpj"]
Oct 30 18:40:43.315229 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:43.315034    4924 cpu_manager.go:414] "containerMap: removing container" podUID="be409e3c-3420-4250-bfa7-3b01ecd69d9b" containerName="registry"
Oct 30 18:40:43.315229 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:43.315044    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be409e3c-3420-4250-bfa7-3b01ecd69d9b" containerName="registry"
Oct 30 18:40:43.315229 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:43.315076    4924 cpu_manager.go:414] "containerMap: removing container" podUID="92981ab7-a41b-482f-98c3-a328e5e8d6f0" containerName="registry"
Oct 30 18:40:43.315229 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:43.315081    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92981ab7-a41b-482f-98c3-a328e5e8d6f0" containerName="registry"
Oct 30 18:40:43.315229 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:43.315139    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="be409e3c-3420-4250-bfa7-3b01ecd69d9b" containerName="registry"
Oct 30 18:40:43.315229 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:43.315149    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="92981ab7-a41b-482f-98c3-a328e5e8d6f0" containerName="registry"
Oct 30 18:40:43.317959 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:43.317937    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zmwpj"
Oct 30 18:40:43.367797 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:43.367771    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 18:40:43.414246 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:43.414218    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 18:40:43.415126 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:43.415109    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-zmwpj"]
Oct 30 18:40:43.415374 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:43.415353    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz8xw\" (UniqueName: \"kubernetes.io/projected/e7a6031a-754e-44d2-8c29-f882a7fa8c92-kube-api-access-kz8xw\") pod \"cert-manager-webhook-5dbbbf75cc-zmwpj\" (UID: \"e7a6031a-754e-44d2-8c29-f882a7fa8c92\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zmwpj"
Oct 30 18:40:43.415416 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:43.415404    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7a6031a-754e-44d2-8c29-f882a7fa8c92-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-zmwpj\" (UID: \"e7a6031a-754e-44d2-8c29-f882a7fa8c92\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zmwpj"
Oct 30 18:40:43.460906 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:43.460885    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-6snk7\""
Oct 30 18:40:43.515902 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:43.515879    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7a6031a-754e-44d2-8c29-f882a7fa8c92-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-zmwpj\" (UID: \"e7a6031a-754e-44d2-8c29-f882a7fa8c92\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zmwpj"
Oct 30 18:40:43.516009 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:43.515925    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz8xw\" (UniqueName: \"kubernetes.io/projected/e7a6031a-754e-44d2-8c29-f882a7fa8c92-kube-api-access-kz8xw\") pod \"cert-manager-webhook-5dbbbf75cc-zmwpj\" (UID: \"e7a6031a-754e-44d2-8c29-f882a7fa8c92\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zmwpj"
Oct 30 18:40:43.570298 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:43.570248    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7a6031a-754e-44d2-8c29-f882a7fa8c92-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-zmwpj\" (UID: \"e7a6031a-754e-44d2-8c29-f882a7fa8c92\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zmwpj"
Oct 30 18:40:43.570404 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:43.570387    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz8xw\" (UniqueName: \"kubernetes.io/projected/e7a6031a-754e-44d2-8c29-f882a7fa8c92-kube-api-access-kz8xw\") pod \"cert-manager-webhook-5dbbbf75cc-zmwpj\" (UID: \"e7a6031a-754e-44d2-8c29-f882a7fa8c92\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zmwpj"
Oct 30 18:40:43.644472 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:43.644445    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zmwpj"
Oct 30 18:40:44.015635 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:44.015462    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-zmwpj"]
Oct 30 18:40:44.018322 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:40:44.018295    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7a6031a_754e_44d2_8c29_f882a7fa8c92.slice/crio-6bfedee5bb644a4062509c4fac3c9d0e0c6783999330344fb35ed13a82d6bc8c WatchSource:0}: Error finding container 6bfedee5bb644a4062509c4fac3c9d0e0c6783999330344fb35ed13a82d6bc8c: Status 404 returned error can't find the container with id 6bfedee5bb644a4062509c4fac3c9d0e0c6783999330344fb35ed13a82d6bc8c
Oct 30 18:40:44.794134 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:44.794097    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zmwpj" event={"ID":"e7a6031a-754e-44d2-8c29-f882a7fa8c92","Type":"ContainerStarted","Data":"6bfedee5bb644a4062509c4fac3c9d0e0c6783999330344fb35ed13a82d6bc8c"}
Oct 30 18:40:45.712521 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:40:45.712470    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:40:46.526536 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:46.526497    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:40:49.414196 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:49.414161    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:40:49.610974 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:49.610938    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:40:49.827361 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:49.827278    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zmwpj" event={"ID":"e7a6031a-754e-44d2-8c29-f882a7fa8c92","Type":"ContainerStarted","Data":"f65f270fdfff7335122dee08b139b02df919e0972867476734a7465da174f973"}
Oct 30 18:40:49.827503 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:49.827417    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zmwpj"
Oct 30 18:40:51.381164 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:51.381112    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zmwpj" podStartSLOduration=3.54501098 podStartE2EDuration="8.381097148s" totalImagesPullingTime="4.836086168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:49.93639845 +0000 UTC m=+251.213448895" watchObservedRunningTime="2026-10-30 18:40:51.381097148 +0000 UTC m=+252.658147593"
Oct 30 18:40:51.381987 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:51.381969    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-lxfl6"]
Oct 30 18:40:51.385039 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:51.385023    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-lxfl6"
Oct 30 18:40:51.477766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:51.477730    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b036561-94cf-4ceb-aaaf-9a8dc53aceb8-tmp\") pod \"openshift-lws-operator-8465994787-lxfl6\" (UID: \"0b036561-94cf-4ceb-aaaf-9a8dc53aceb8\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-lxfl6"
Oct 30 18:40:51.477921 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:51.477782    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clhzk\" (UniqueName: \"kubernetes.io/projected/0b036561-94cf-4ceb-aaaf-9a8dc53aceb8-kube-api-access-clhzk\") pod \"openshift-lws-operator-8465994787-lxfl6\" (UID: \"0b036561-94cf-4ceb-aaaf-9a8dc53aceb8\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-lxfl6"
Oct 30 18:40:51.482226 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:51.482207    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:40:51.482420 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:51.482409    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 18:40:51.482498 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:51.482487    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-pjn2r\""
Oct 30 18:40:51.483608 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:51.483591    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-lxfl6"]
Oct 30 18:40:51.578505 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:51.578444    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clhzk\" (UniqueName: \"kubernetes.io/projected/0b036561-94cf-4ceb-aaaf-9a8dc53aceb8-kube-api-access-clhzk\") pod \"openshift-lws-operator-8465994787-lxfl6\" (UID: \"0b036561-94cf-4ceb-aaaf-9a8dc53aceb8\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-lxfl6"
Oct 30 18:40:51.578679 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:51.578550    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b036561-94cf-4ceb-aaaf-9a8dc53aceb8-tmp\") pod \"openshift-lws-operator-8465994787-lxfl6\" (UID: \"0b036561-94cf-4ceb-aaaf-9a8dc53aceb8\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-lxfl6"
Oct 30 18:40:51.578909 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:51.578892    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b036561-94cf-4ceb-aaaf-9a8dc53aceb8-tmp\") pod \"openshift-lws-operator-8465994787-lxfl6\" (UID: \"0b036561-94cf-4ceb-aaaf-9a8dc53aceb8\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-lxfl6"
Oct 30 18:40:51.632822 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:51.632755    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clhzk\" (UniqueName: \"kubernetes.io/projected/0b036561-94cf-4ceb-aaaf-9a8dc53aceb8-kube-api-access-clhzk\") pod \"openshift-lws-operator-8465994787-lxfl6\" (UID: \"0b036561-94cf-4ceb-aaaf-9a8dc53aceb8\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-lxfl6"
Oct 30 18:40:51.695801 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:51.695771    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-lxfl6"
Oct 30 18:40:52.054308 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:52.054223    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-lxfl6"]
Oct 30 18:40:52.056909 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:40:52.056880    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b036561_94cf_4ceb_aaaf_9a8dc53aceb8.slice/crio-18f1934703b2eb475c1465282f323dfd21a5d641100d361bf63da4f73fbae326 WatchSource:0}: Error finding container 18f1934703b2eb475c1465282f323dfd21a5d641100d361bf63da4f73fbae326: Status 404 returned error can't find the container with id 18f1934703b2eb475c1465282f323dfd21a5d641100d361bf63da4f73fbae326
Oct 30 18:40:52.839595 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:52.839561    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-lxfl6" event={"ID":"0b036561-94cf-4ceb-aaaf-9a8dc53aceb8","Type":"ContainerStarted","Data":"18f1934703b2eb475c1465282f323dfd21a5d641100d361bf63da4f73fbae326"}
Oct 30 18:40:55.834101 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:55.834052    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zmwpj"
Oct 30 18:40:56.856262 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:56.856226    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-lxfl6" event={"ID":"0b036561-94cf-4ceb-aaaf-9a8dc53aceb8","Type":"ContainerStarted","Data":"ae2ddee30aa8148a141d15f6708ce4ed9b585cbdecb257541aa67f36e4705509"}
Oct 30 18:40:56.972996 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:56.972949    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-lxfl6" podStartSLOduration=1.904748949 podStartE2EDuration="5.972937119s" totalImagesPullingTime="4.06818817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:56.97244883 +0000 UTC m=+258.249499274" watchObservedRunningTime="2026-10-30 18:40:56.972937119 +0000 UTC m=+258.249987558"
Oct 30 18:40:58.051813 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:58.051770    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:40:58.237774 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:58.237732    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:40:59.917397 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:40:59.917350    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:14.592280 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:14.592249    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:15.529459 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.529429    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2"]
Oct 30 18:41:15.532888 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.532873    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2"
Oct 30 18:41:15.661762 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.661730    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 18:41:15.664105 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.664072    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c-webhook-cert\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2\" (UID: \"9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2"
Oct 30 18:41:15.664245 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.664122    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c-tmp\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2\" (UID: \"9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2"
Oct 30 18:41:15.664320 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.664246    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2\" (UID: \"9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2"
Oct 30 18:41:15.664320 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.664300    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c-metrics-cert\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2\" (UID: \"9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2"
Oct 30 18:41:15.664403 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.664328    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwrdj\" (UniqueName: \"kubernetes.io/projected/9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c-kube-api-access-dwrdj\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2\" (UID: \"9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2"
Oct 30 18:41:15.664464 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.664443    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c-manifests\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2\" (UID: \"9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2"
Oct 30 18:41:15.700605 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.700568    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 18:41:15.700747 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.700613    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 18:41:15.700747 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.700636    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 18:41:15.700747 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.700665    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-t8d96\""
Oct 30 18:41:15.700747 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.700713    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 18:41:15.710953 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:41:15.710925    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:41:15.765503 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.765478    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c-metrics-cert\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2\" (UID: \"9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2"
Oct 30 18:41:15.765628 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.765511    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwrdj\" (UniqueName: \"kubernetes.io/projected/9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c-kube-api-access-dwrdj\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2\" (UID: \"9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2"
Oct 30 18:41:15.765628 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.765536    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c-manifests\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2\" (UID: \"9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2"
Oct 30 18:41:15.765628 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.765554    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c-webhook-cert\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2\" (UID: \"9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2"
Oct 30 18:41:15.765628 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.765578    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c-tmp\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2\" (UID: \"9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2"
Oct 30 18:41:15.765628 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.765613    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2\" (UID: \"9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2"
Oct 30 18:41:15.765980 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.765953    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c-manifests\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2\" (UID: \"9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2"
Oct 30 18:41:15.767808 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.767781    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c-tmp\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2\" (UID: \"9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2"
Oct 30 18:41:15.768235 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.768208    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c-metrics-cert\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2\" (UID: \"9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2"
Oct 30 18:41:15.768379 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.768288    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c-webhook-cert\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2\" (UID: \"9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2"
Oct 30 18:41:15.768379 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.768353    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2\" (UID: \"9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2"
Oct 30 18:41:15.769703 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.769682    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2"]
Oct 30 18:41:15.822255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.822202    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwrdj\" (UniqueName: \"kubernetes.io/projected/9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c-kube-api-access-dwrdj\") pod \"opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2\" (UID: \"9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c\") " pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2"
Oct 30 18:41:15.847783 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:15.847763    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2"
Oct 30 18:41:16.115605 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:16.115577    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:16.323899 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:16.323828    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2"]
Oct 30 18:41:16.325471 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:41:16.325443    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c72e2bb_50ca_4baf_9991_7cf8c7f80f1c.slice/crio-9f2782c1cee3172aa83c24e9ed5ae17c63214c95d6d8bb329b069f245d55327c WatchSource:0}: Error finding container 9f2782c1cee3172aa83c24e9ed5ae17c63214c95d6d8bb329b069f245d55327c: Status 404 returned error can't find the container with id 9f2782c1cee3172aa83c24e9ed5ae17c63214c95d6d8bb329b069f245d55327c
Oct 30 18:41:16.877857 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:16.877817    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:16.935734 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:16.935704    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2" event={"ID":"9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c","Type":"ContainerStarted","Data":"9f2782c1cee3172aa83c24e9ed5ae17c63214c95d6d8bb329b069f245d55327c"}
Oct 30 18:41:17.084620 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:17.084586    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:20.916494 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:20.916450    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:20.955200 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:20.955173    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2" event={"ID":"9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c","Type":"ContainerStarted","Data":"26dae9348ddfe1a31f24035c3a7d34eb6cc650cbe9328146311f4dc0286b7eb4"}
Oct 30 18:41:21.101932 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:21.101900    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:21.349131 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:21.349090    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:21.960479 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:21.960444    4924 generic.go:356] "Generic (PLEG): container finished" podID="9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c" containerID="26dae9348ddfe1a31f24035c3a7d34eb6cc650cbe9328146311f4dc0286b7eb4" exitCode=0
Oct 30 18:41:21.960888 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:21.960525    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2" event={"ID":"9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c","Type":"ContainerDied","Data":"26dae9348ddfe1a31f24035c3a7d34eb6cc650cbe9328146311f4dc0286b7eb4"}
Oct 30 18:41:22.966326 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:22.966292    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2" event={"ID":"9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c","Type":"ContainerStarted","Data":"0bf8b398834d54187bc0e2947af0ab4d1d12344de8b79a83c20e95817160ad91"}
Oct 30 18:41:22.966695 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:22.966427    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2"
Oct 30 18:41:23.136121 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:23.136039    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2" podStartSLOduration=2.604565977 podStartE2EDuration="8.13602204s" totalImagesPullingTime="5.082115469s" totalInitContainerRuntime="449.340594ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:23.089426503 +0000 UTC m=+284.366476947" watchObservedRunningTime="2026-10-30 18:41:23.13602204 +0000 UTC m=+284.413072477"
Oct 30 18:41:29.896147 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:29.896111    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:33.973034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:33.973002    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2"
Oct 30 18:41:37.627088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:37.627024    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:37.819537 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:37.819502    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:38.849641 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:38.849614    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 18:41:38.851284 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:38.851262    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 18:41:38.927190 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:38.927164    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-58d577dc79-xhbrt"]
Oct 30 18:41:38.931820 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:38.931793    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:38.933407 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:38.933389    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58d577dc79-xhbrt"
Oct 30 18:41:38.986818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:38.986791    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58d577dc79-xhbrt"]
Oct 30 18:41:38.991702 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:38.991682    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 18:41:39.032444 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:39.032409    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 18:41:39.032444 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:39.032430    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5dblg\""
Oct 30 18:41:39.044387 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:39.044366    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/630856d6-3139-44fd-958f-8d031ffb8f9a-tmp\") pod \"kube-auth-proxy-58d577dc79-xhbrt\" (UID: \"630856d6-3139-44fd-958f-8d031ffb8f9a\") " pod="openshift-ingress/kube-auth-proxy-58d577dc79-xhbrt"
Oct 30 18:41:39.044480 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:39.044405    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d52hz\" (UniqueName: \"kubernetes.io/projected/630856d6-3139-44fd-958f-8d031ffb8f9a-kube-api-access-d52hz\") pod \"kube-auth-proxy-58d577dc79-xhbrt\" (UID: \"630856d6-3139-44fd-958f-8d031ffb8f9a\") " pod="openshift-ingress/kube-auth-proxy-58d577dc79-xhbrt"
Oct 30 18:41:39.044534 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:39.044486    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630856d6-3139-44fd-958f-8d031ffb8f9a-tls-certs\") pod \"kube-auth-proxy-58d577dc79-xhbrt\" (UID: \"630856d6-3139-44fd-958f-8d031ffb8f9a\") " pod="openshift-ingress/kube-auth-proxy-58d577dc79-xhbrt"
Oct 30 18:41:39.126000 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:39.125919    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:39.145697 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:39.145673    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630856d6-3139-44fd-958f-8d031ffb8f9a-tls-certs\") pod \"kube-auth-proxy-58d577dc79-xhbrt\" (UID: \"630856d6-3139-44fd-958f-8d031ffb8f9a\") " pod="openshift-ingress/kube-auth-proxy-58d577dc79-xhbrt"
Oct 30 18:41:39.145770 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:39.145717    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/630856d6-3139-44fd-958f-8d031ffb8f9a-tmp\") pod \"kube-auth-proxy-58d577dc79-xhbrt\" (UID: \"630856d6-3139-44fd-958f-8d031ffb8f9a\") " pod="openshift-ingress/kube-auth-proxy-58d577dc79-xhbrt"
Oct 30 18:41:39.145857 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:39.145840    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d52hz\" (UniqueName: \"kubernetes.io/projected/630856d6-3139-44fd-958f-8d031ffb8f9a-kube-api-access-d52hz\") pod \"kube-auth-proxy-58d577dc79-xhbrt\" (UID: \"630856d6-3139-44fd-958f-8d031ffb8f9a\") " pod="openshift-ingress/kube-auth-proxy-58d577dc79-xhbrt"
Oct 30 18:41:39.147933 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:39.147908    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/630856d6-3139-44fd-958f-8d031ffb8f9a-tmp\") pod \"kube-auth-proxy-58d577dc79-xhbrt\" (UID: \"630856d6-3139-44fd-958f-8d031ffb8f9a\") " pod="openshift-ingress/kube-auth-proxy-58d577dc79-xhbrt"
Oct 30 18:41:39.148203 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:39.148185    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/630856d6-3139-44fd-958f-8d031ffb8f9a-tls-certs\") pod \"kube-auth-proxy-58d577dc79-xhbrt\" (UID: \"630856d6-3139-44fd-958f-8d031ffb8f9a\") " pod="openshift-ingress/kube-auth-proxy-58d577dc79-xhbrt"
Oct 30 18:41:39.200668 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:39.200639    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d52hz\" (UniqueName: \"kubernetes.io/projected/630856d6-3139-44fd-958f-8d031ffb8f9a-kube-api-access-d52hz\") pod \"kube-auth-proxy-58d577dc79-xhbrt\" (UID: \"630856d6-3139-44fd-958f-8d031ffb8f9a\") " pod="openshift-ingress/kube-auth-proxy-58d577dc79-xhbrt"
Oct 30 18:41:39.243555 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:39.243533    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58d577dc79-xhbrt"
Oct 30 18:41:39.607666 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:39.607637    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58d577dc79-xhbrt"]
Oct 30 18:41:39.610508 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:41:39.610486    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630856d6_3139_44fd_958f_8d031ffb8f9a.slice/crio-51ab2a82507f9644db09152700b0e11431b7b0bad3741db59af55bd064f803ed WatchSource:0}: Error finding container 51ab2a82507f9644db09152700b0e11431b7b0bad3741db59af55bd064f803ed: Status 404 returned error can't find the container with id 51ab2a82507f9644db09152700b0e11431b7b0bad3741db59af55bd064f803ed
Oct 30 18:41:40.038548 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:40.038461    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58d577dc79-xhbrt" event={"ID":"630856d6-3139-44fd-958f-8d031ffb8f9a","Type":"ContainerStarted","Data":"51ab2a82507f9644db09152700b0e11431b7b0bad3741db59af55bd064f803ed"}
Oct 30 18:41:40.427202 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:40.427159    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:40.611839 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:40.611807    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:41.519504 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:41.519465    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:43.054313 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:43.054275    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58d577dc79-xhbrt" event={"ID":"630856d6-3139-44fd-958f-8d031ffb8f9a","Type":"ContainerStarted","Data":"c5ec9ac8815a3b09815c0466a3013a9e45ab375812227cc74299e8d5b97f236f"}
Oct 30 18:41:43.165252 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:43.165201    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-58d577dc79-xhbrt" podStartSLOduration=2.277873555 podStartE2EDuration="5.165187076s" totalImagesPullingTime="2.887313521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:43.163986673 +0000 UTC m=+304.441037118" watchObservedRunningTime="2026-10-30 18:41:43.165187076 +0000 UTC m=+304.442237520"
Oct 30 18:41:43.172425 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:43.172398    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:43.357351 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:43.357308    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:44.705105 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:44.705057    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:45.710143 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:41:45.710111    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:41:46.378917 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:46.378889    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:46.602620 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:46.602590    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:47.540686 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:47.540657    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:47.737275 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:47.737244    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:47.918564 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:47.918528    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:48.106262 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:48.106216    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:48.308794 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:48.308714    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:48.547454 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:48.547425    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:49.469835 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:49.469803    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:41:50.254140 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:50.254110    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b9b666444-lm589"]
Oct 30 18:41:50.257309 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:50.257288    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7b9b666444-lm589"
Oct 30 18:41:50.307163 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:50.307134    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-whgwz\""
Oct 30 18:41:50.334536 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:50.334506    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e732d7ef-74cb-42f4-83cc-4c00a3c4b684-metrics-cert\") pod \"lws-controller-manager-7b9b666444-lm589\" (UID: \"e732d7ef-74cb-42f4-83cc-4c00a3c4b684\") " pod="openshift-lws-operator/lws-controller-manager-7b9b666444-lm589"
Oct 30 18:41:50.334673 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:50.334548    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e732d7ef-74cb-42f4-83cc-4c00a3c4b684-cert\") pod \"lws-controller-manager-7b9b666444-lm589\" (UID: \"e732d7ef-74cb-42f4-83cc-4c00a3c4b684\") " pod="openshift-lws-operator/lws-controller-manager-7b9b666444-lm589"
Oct 30 18:41:50.334673 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:50.334572    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e732d7ef-74cb-42f4-83cc-4c00a3c4b684-manager-config\") pod \"lws-controller-manager-7b9b666444-lm589\" (UID: \"e732d7ef-74cb-42f4-83cc-4c00a3c4b684\") " pod="openshift-lws-operator/lws-controller-manager-7b9b666444-lm589"
Oct 30 18:41:50.334744 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:50.334677    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmp8q\" (UniqueName: \"kubernetes.io/projected/e732d7ef-74cb-42f4-83cc-4c00a3c4b684-kube-api-access-zmp8q\") pod \"lws-controller-manager-7b9b666444-lm589\" (UID: \"e732d7ef-74cb-42f4-83cc-4c00a3c4b684\") " pod="openshift-lws-operator/lws-controller-manager-7b9b666444-lm589"
Oct 30 18:41:50.353993 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:50.353967    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 30 18:41:50.354961 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:50.354940    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b9b666444-lm589"]
Oct 30 18:41:50.398362 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:50.398333    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 30 18:41:50.398522 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:50.398442    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 30 18:41:50.435229 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:50.435195    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmp8q\" (UniqueName: \"kubernetes.io/projected/e732d7ef-74cb-42f4-83cc-4c00a3c4b684-kube-api-access-zmp8q\") pod \"lws-controller-manager-7b9b666444-lm589\" (UID: \"e732d7ef-74cb-42f4-83cc-4c00a3c4b684\") " pod="openshift-lws-operator/lws-controller-manager-7b9b666444-lm589"
Oct 30 18:41:50.435229 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:50.435234    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e732d7ef-74cb-42f4-83cc-4c00a3c4b684-metrics-cert\") pod \"lws-controller-manager-7b9b666444-lm589\" (UID: \"e732d7ef-74cb-42f4-83cc-4c00a3c4b684\") " pod="openshift-lws-operator/lws-controller-manager-7b9b666444-lm589"
Oct 30 18:41:50.435442 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:50.435262    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e732d7ef-74cb-42f4-83cc-4c00a3c4b684-cert\") pod \"lws-controller-manager-7b9b666444-lm589\" (UID: \"e732d7ef-74cb-42f4-83cc-4c00a3c4b684\") " pod="openshift-lws-operator/lws-controller-manager-7b9b666444-lm589"
Oct 30 18:41:50.435442 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:50.435291    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e732d7ef-74cb-42f4-83cc-4c00a3c4b684-manager-config\") pod \"lws-controller-manager-7b9b666444-lm589\" (UID: \"e732d7ef-74cb-42f4-83cc-4c00a3c4b684\") " pod="openshift-lws-operator/lws-controller-manager-7b9b666444-lm589"
Oct 30 18:41:50.436045 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:50.436023    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e732d7ef-74cb-42f4-83cc-4c00a3c4b684-manager-config\") pod \"lws-controller-manager-7b9b666444-lm589\" (UID: \"e732d7ef-74cb-42f4-83cc-4c00a3c4b684\") " pod="openshift-lws-operator/lws-controller-manager-7b9b666444-lm589"
Oct 30 18:41:50.437855 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:50.437828    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e732d7ef-74cb-42f4-83cc-4c00a3c4b684-metrics-cert\") pod \"lws-controller-manager-7b9b666444-lm589\" (UID: \"e732d7ef-74cb-42f4-83cc-4c00a3c4b684\") " pod="openshift-lws-operator/lws-controller-manager-7b9b666444-lm589"
Oct 30 18:41:50.437943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:50.437829    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e732d7ef-74cb-42f4-83cc-4c00a3c4b684-cert\") pod \"lws-controller-manager-7b9b666444-lm589\" (UID: \"e732d7ef-74cb-42f4-83cc-4c00a3c4b684\") " pod="openshift-lws-operator/lws-controller-manager-7b9b666444-lm589"
Oct 30 18:41:50.489581 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:50.489547    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmp8q\" (UniqueName: \"kubernetes.io/projected/e732d7ef-74cb-42f4-83cc-4c00a3c4b684-kube-api-access-zmp8q\") pod \"lws-controller-manager-7b9b666444-lm589\" (UID: \"e732d7ef-74cb-42f4-83cc-4c00a3c4b684\") " pod="openshift-lws-operator/lws-controller-manager-7b9b666444-lm589"
Oct 30 18:41:50.568115 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:50.568007    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7b9b666444-lm589"
Oct 30 18:41:50.927663 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:50.927640    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b9b666444-lm589"]
Oct 30 18:41:50.930154 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:41:50.930116    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode732d7ef_74cb_42f4_83cc_4c00a3c4b684.slice/crio-31f7d6930b9875ccf38f3eb57ef4d42edf7a76e0ab1c47dfec563c7bbd4bf56a WatchSource:0}: Error finding container 31f7d6930b9875ccf38f3eb57ef4d42edf7a76e0ab1c47dfec563c7bbd4bf56a: Status 404 returned error can't find the container with id 31f7d6930b9875ccf38f3eb57ef4d42edf7a76e0ab1c47dfec563c7bbd4bf56a
Oct 30 18:41:50.932170 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:50.932150    4924 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:41:51.086966 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:51.086928    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7b9b666444-lm589" event={"ID":"e732d7ef-74cb-42f4-83cc-4c00a3c4b684","Type":"ContainerStarted","Data":"31f7d6930b9875ccf38f3eb57ef4d42edf7a76e0ab1c47dfec563c7bbd4bf56a"}
Oct 30 18:41:55.106660 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:55.106622    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7b9b666444-lm589" event={"ID":"e732d7ef-74cb-42f4-83cc-4c00a3c4b684","Type":"ContainerStarted","Data":"d586946f2f958ba935b3832b095e9812bca1b60c39b51b01059d530a686d92af"}
Oct 30 18:41:55.107083 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:55.106757    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7b9b666444-lm589"
Oct 30 18:41:55.215562 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:41:55.215506    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7b9b666444-lm589" podStartSLOduration=1.6651907750000001 podStartE2EDuration="5.215492493s" totalImagesPullingTime="3.550301718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:55.21437931 +0000 UTC m=+316.491429754" watchObservedRunningTime="2026-10-30 18:41:55.215492493 +0000 UTC m=+316.492542938"
Oct 30 18:42:05.906392 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:42:05.906352    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:42:06.114045 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:42:06.114008    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7b9b666444-lm589"
Oct 30 18:42:06.428311 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:42:06.428260    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:42:15.709895 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:42:15.709860    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:42:30.232228 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:42:30.232194    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:42:45.710516 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:42:45.710480    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:43:02.145556 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:43:02.145523    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:02.322568 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:43:02.322533    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:02.503870 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:43:02.503792    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:02.675586 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:43:02.675555    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:02.854411 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:43:02.854381    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:03.050222 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:43:03.050188    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:03.248531 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:43:03.248445    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:04.285573 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:43:04.285536    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:09.102011 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:43:09.101964    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:09.289764 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:43:09.289728    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:09.514348 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:43:09.514250    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:09.698493 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:43:09.698459    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:09.886924 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:43:09.886892    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:10.102014 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:43:10.101982    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:10.283943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:43:10.283858    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:15.709689 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:43:15.709652    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:43:30.554403 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:43:30.554312    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:43:38.842645 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:43:38.842615    4924 scope.go:122] "RemoveContainer" containerID="da9398581e049624eff438d8fc0366d75919657cea7ce3bbcd953f8e68c9c6d3"
Oct 30 18:43:45.709267 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:43:45.709219    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:44:11.241664 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:11.241628    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:11.469717 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:11.469681    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:15.710093 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:44:15.710045    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:44:26.676315 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:26.676267    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:27.455479 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:27.455447    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:28.257681 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:28.257646    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:30.885031 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:30.885000    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:31.265800 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:31.265704    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:31.514977 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:31.514927    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:31.747401 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:31.747369    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:33.115452 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:33.115413    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:34.262658 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:34.262623    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:35.276192 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:35.276159    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:36.461725 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:36.461692    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:37.929932 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:37.929881    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:41.871370 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:41.871337    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:42.050100 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:42.050054    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:42.269883 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:42.269786    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:42.477790 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:42.477750    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:42.720484 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:42.720446    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:42.902898 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:42.902854    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:44.822975 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:44.822941    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-lpgm6"]
Oct 30 18:44:44.826232 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:44.826211    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lpgm6"
Oct 30 18:44:44.843003 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:44.842976    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:44.875965 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:44.875945    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 18:44:44.902704 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:44.902680    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/63b4b347-6aee-4bb6-80e7-7103bf398747-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-lpgm6\" (UID: \"63b4b347-6aee-4bb6-80e7-7103bf398747\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lpgm6"
Oct 30 18:44:44.902804 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:44.902746    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr7c7\" (UniqueName: \"kubernetes.io/projected/63b4b347-6aee-4bb6-80e7-7103bf398747-kube-api-access-jr7c7\") pod \"kuadrant-console-plugin-66c4fb575d-lpgm6\" (UID: \"63b4b347-6aee-4bb6-80e7-7103bf398747\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lpgm6"
Oct 30 18:44:44.902854 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:44.902804    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/63b4b347-6aee-4bb6-80e7-7103bf398747-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-lpgm6\" (UID: \"63b4b347-6aee-4bb6-80e7-7103bf398747\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lpgm6"
Oct 30 18:44:44.921997 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:44.921973    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 18:44:44.922118 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:44.922095    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 18:44:44.922172 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:44.922114    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 18:44:44.923162 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:44.923141    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-lpgm6"]
Oct 30 18:44:44.976766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:44.976746    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-szvzq\""
Oct 30 18:44:45.004070 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:45.004049    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7c7\" (UniqueName: \"kubernetes.io/projected/63b4b347-6aee-4bb6-80e7-7103bf398747-kube-api-access-jr7c7\") pod \"kuadrant-console-plugin-66c4fb575d-lpgm6\" (UID: \"63b4b347-6aee-4bb6-80e7-7103bf398747\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lpgm6"
Oct 30 18:44:45.004147 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:45.004089    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/63b4b347-6aee-4bb6-80e7-7103bf398747-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-lpgm6\" (UID: \"63b4b347-6aee-4bb6-80e7-7103bf398747\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lpgm6"
Oct 30 18:44:45.004147 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:45.004127    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/63b4b347-6aee-4bb6-80e7-7103bf398747-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-lpgm6\" (UID: \"63b4b347-6aee-4bb6-80e7-7103bf398747\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lpgm6"
Oct 30 18:44:45.004830 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:45.004813    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/63b4b347-6aee-4bb6-80e7-7103bf398747-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-lpgm6\" (UID: \"63b4b347-6aee-4bb6-80e7-7103bf398747\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lpgm6"
Oct 30 18:44:45.006501 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:45.006481    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/63b4b347-6aee-4bb6-80e7-7103bf398747-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-lpgm6\" (UID: \"63b4b347-6aee-4bb6-80e7-7103bf398747\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lpgm6"
Oct 30 18:44:45.058995 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:45.058964    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr7c7\" (UniqueName: \"kubernetes.io/projected/63b4b347-6aee-4bb6-80e7-7103bf398747-kube-api-access-jr7c7\") pod \"kuadrant-console-plugin-66c4fb575d-lpgm6\" (UID: \"63b4b347-6aee-4bb6-80e7-7103bf398747\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lpgm6"
Oct 30 18:44:45.060025 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:45.060004    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:45.139402 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:45.139376    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lpgm6"
Oct 30 18:44:45.498393 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:45.498370    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-lpgm6"]
Oct 30 18:44:45.500536 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:44:45.500505    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63b4b347_6aee_4bb6_80e7_7103bf398747.slice/crio-33631f381c690d7a16152076b529d9ac0db4be796102d585986c710cf4a8648c WatchSource:0}: Error finding container 33631f381c690d7a16152076b529d9ac0db4be796102d585986c710cf4a8648c: Status 404 returned error can't find the container with id 33631f381c690d7a16152076b529d9ac0db4be796102d585986c710cf4a8648c
Oct 30 18:44:45.710206 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:44:45.710169    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:44:45.776983 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:45.776903    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lpgm6" event={"ID":"63b4b347-6aee-4bb6-80e7-7103bf398747","Type":"ContainerStarted","Data":"33631f381c690d7a16152076b529d9ac0db4be796102d585986c710cf4a8648c"}
Oct 30 18:44:50.678169 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:50.678086    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:50.956148 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:50.956056    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:51.807569 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:51.807533    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lpgm6" event={"ID":"63b4b347-6aee-4bb6-80e7-7103bf398747","Type":"ContainerStarted","Data":"6022ceafca0feccc1719d369309b360a2b5bdc6c6522183258f351cbf2ae5e59"}
Oct 30 18:44:51.921000 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:51.920948    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-lpgm6" podStartSLOduration=2.377439386 podStartE2EDuration="7.920934025s" totalImagesPullingTime="5.543494639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:51.91972519 +0000 UTC m=+493.196775635" watchObservedRunningTime="2026-10-30 18:44:51.920934025 +0000 UTC m=+493.197984469"
Oct 30 18:44:52.704999 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:52.704962    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:53.689764 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:53.689703    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:53.893901 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:53.893858    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:56.151382 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:56.151350    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:44:56.361884 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:44:56.361848    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:03.089516 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:03.089476    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:03.281005 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:03.280973    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:03.484738 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:03.484709    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:03.673033 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:03.673000    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:03.856922 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:03.856890    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:12.547691 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:12.547656    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:12.748264 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:12.748230    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:15.501325 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:15.501255    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:15.677569 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:15.677532    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:15.710282 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:45:15.710250    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:45:17.386421 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:17.386387    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:17.945235 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:17.945199    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:18.121491 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:18.121444    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:26.490113 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:26.490059    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:26.787240 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:26.787141    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:31.203960 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:31.203924    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:45.710082 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:45:45.710037    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:45:47.118364 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:47.118332    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:47.348452 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:47.348408    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:47.558470 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:47.558360    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:48.298522 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:48.298487    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-d5hzm"]
Oct 30 18:45:48.301305 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:48.301288    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-d5hzm"
Oct 30 18:45:48.350789 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:48.350764    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-w8f7v\""
Oct 30 18:45:48.398143 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:48.398114    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-d5hzm"]
Oct 30 18:45:48.411547 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:48.411520    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq42g\" (UniqueName: \"kubernetes.io/projected/84fb351f-ede4-4f1d-a3ee-84565973a447-kube-api-access-lq42g\") pod \"authorino-57f7f8659f-d5hzm\" (UID: \"84fb351f-ede4-4f1d-a3ee-84565973a447\") " pod="kuadrant-system/authorino-57f7f8659f-d5hzm"
Oct 30 18:45:48.504809 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:48.504785    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:48.512860 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:48.512839    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq42g\" (UniqueName: \"kubernetes.io/projected/84fb351f-ede4-4f1d-a3ee-84565973a447-kube-api-access-lq42g\") pod \"authorino-57f7f8659f-d5hzm\" (UID: \"84fb351f-ede4-4f1d-a3ee-84565973a447\") " pod="kuadrant-system/authorino-57f7f8659f-d5hzm"
Oct 30 18:45:48.568962 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:48.568911    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq42g\" (UniqueName: \"kubernetes.io/projected/84fb351f-ede4-4f1d-a3ee-84565973a447-kube-api-access-lq42g\") pod \"authorino-57f7f8659f-d5hzm\" (UID: \"84fb351f-ede4-4f1d-a3ee-84565973a447\") " pod="kuadrant-system/authorino-57f7f8659f-d5hzm"
Oct 30 18:45:48.611857 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:48.611832    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-d5hzm"
Oct 30 18:45:48.690084 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:48.690040    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:45:48.969970 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:48.969858    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-d5hzm"]
Oct 30 18:45:48.972573 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:45:48.972545    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84fb351f_ede4_4f1d_a3ee_84565973a447.slice/crio-8d5faad3a902f941413f5c4ef6e391f66e2ea197577df0792b3f1e479bc9fd04 WatchSource:0}: Error finding container 8d5faad3a902f941413f5c4ef6e391f66e2ea197577df0792b3f1e479bc9fd04: Status 404 returned error can't find the container with id 8d5faad3a902f941413f5c4ef6e391f66e2ea197577df0792b3f1e479bc9fd04
Oct 30 18:45:49.024467 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:49.024445    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-d5hzm" event={"ID":"84fb351f-ede4-4f1d-a3ee-84565973a447","Type":"ContainerStarted","Data":"8d5faad3a902f941413f5c4ef6e391f66e2ea197577df0792b3f1e479bc9fd04"}
Oct 30 18:45:50.023981 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:50.023945    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-b8gz8"]
Oct 30 18:45:50.027700 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:50.027675    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-b8gz8"
Oct 30 18:45:50.081767 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:50.081741    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-b8gz8"]
Oct 30 18:45:50.124724 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:50.124697    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqnvt\" (UniqueName: \"kubernetes.io/projected/9908a9c0-7387-4e15-bbc0-6d58d7c3fb72-kube-api-access-lqnvt\") pod \"authorino-56c6c9ddd5-b8gz8\" (UID: \"9908a9c0-7387-4e15-bbc0-6d58d7c3fb72\") " pod="kuadrant-system/authorino-56c6c9ddd5-b8gz8"
Oct 30 18:45:50.225365 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:50.225334    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqnvt\" (UniqueName: \"kubernetes.io/projected/9908a9c0-7387-4e15-bbc0-6d58d7c3fb72-kube-api-access-lqnvt\") pod \"authorino-56c6c9ddd5-b8gz8\" (UID: \"9908a9c0-7387-4e15-bbc0-6d58d7c3fb72\") " pod="kuadrant-system/authorino-56c6c9ddd5-b8gz8"
Oct 30 18:45:50.285322 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:50.285248    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqnvt\" (UniqueName: \"kubernetes.io/projected/9908a9c0-7387-4e15-bbc0-6d58d7c3fb72-kube-api-access-lqnvt\") pod \"authorino-56c6c9ddd5-b8gz8\" (UID: \"9908a9c0-7387-4e15-bbc0-6d58d7c3fb72\") " pod="kuadrant-system/authorino-56c6c9ddd5-b8gz8"
Oct 30 18:45:50.339074 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:50.339047    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-b8gz8"
Oct 30 18:45:50.698916 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:50.698881    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-b8gz8"]
Oct 30 18:45:50.704085 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:45:50.703790    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9908a9c0_7387_4e15_bbc0_6d58d7c3fb72.slice/crio-614bea46a1c7ccfe341f9518427c12d746984ab78aa2294b02cd106bc54ef89d WatchSource:0}: Error finding container 614bea46a1c7ccfe341f9518427c12d746984ab78aa2294b02cd106bc54ef89d: Status 404 returned error can't find the container with id 614bea46a1c7ccfe341f9518427c12d746984ab78aa2294b02cd106bc54ef89d
Oct 30 18:45:51.035628 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:51.035538    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-b8gz8" event={"ID":"9908a9c0-7387-4e15-bbc0-6d58d7c3fb72","Type":"ContainerStarted","Data":"614bea46a1c7ccfe341f9518427c12d746984ab78aa2294b02cd106bc54ef89d"}
Oct 30 18:45:53.046593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:53.046513    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-b8gz8" event={"ID":"9908a9c0-7387-4e15-bbc0-6d58d7c3fb72","Type":"ContainerStarted","Data":"f9f2f54a7f4b9151fce2ea937a0cc3efc7fbb75ac96d0165278e7d4363b12c57"}
Oct 30 18:45:53.048486 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:53.048457    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-d5hzm" event={"ID":"84fb351f-ede4-4f1d-a3ee-84565973a447","Type":"ContainerStarted","Data":"2c65b76a5f73c2525728af0f971698db1348ad512360d62f13c173e76097ebeb"}
Oct 30 18:45:53.156545 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:53.156495    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-b8gz8" podStartSLOduration=1.969592017 podStartE2EDuration="4.156482145s" totalImagesPullingTime="2.186890128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:53.156084149 +0000 UTC m=+554.433134597" watchObservedRunningTime="2026-10-30 18:45:53.156482145 +0000 UTC m=+554.433532583"
Oct 30 18:45:53.183097 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:53.183047    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-d5hzm"]
Oct 30 18:45:53.267117 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:53.267049    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-d5hzm" podStartSLOduration=1.544827234 podStartE2EDuration="5.267025254s" totalImagesPullingTime="3.72219802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:53.266428289 +0000 UTC m=+554.543478734" watchObservedRunningTime="2026-10-30 18:45:53.267025254 +0000 UTC m=+554.544075909"
Oct 30 18:45:55.056293 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:55.056233    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-d5hzm" podUID="84fb351f-ede4-4f1d-a3ee-84565973a447" containerName="authorino" containerID="cri-o://2c65b76a5f73c2525728af0f971698db1348ad512360d62f13c173e76097ebeb" gracePeriod=30
Oct 30 18:45:55.288536 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:55.288515    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-d5hzm"
Oct 30 18:45:55.365918 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:55.365894    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84fb351f-ede4-4f1d-a3ee-84565973a447-kube-api-access-lq42g\" (UniqueName: \"kubernetes.io/projected/84fb351f-ede4-4f1d-a3ee-84565973a447-kube-api-access-lq42g\") pod \"84fb351f-ede4-4f1d-a3ee-84565973a447\" (UID: \"84fb351f-ede4-4f1d-a3ee-84565973a447\") "
Oct 30 18:45:55.367947 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:55.367923    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84fb351f-ede4-4f1d-a3ee-84565973a447-kube-api-access-lq42g" pod "84fb351f-ede4-4f1d-a3ee-84565973a447" (UID: "84fb351f-ede4-4f1d-a3ee-84565973a447"). InnerVolumeSpecName "kube-api-access-lq42g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:45:55.467220 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:55.467187    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq42g\" (UniqueName: \"kubernetes.io/projected/84fb351f-ede4-4f1d-a3ee-84565973a447-kube-api-access-lq42g\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:45:56.061018 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:56.060984    4924 generic.go:356] "Generic (PLEG): container finished" podID="84fb351f-ede4-4f1d-a3ee-84565973a447" containerID="2c65b76a5f73c2525728af0f971698db1348ad512360d62f13c173e76097ebeb" exitCode=0
Oct 30 18:45:56.061516 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:56.061039    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-d5hzm"
Oct 30 18:45:56.061516 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:56.061040    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-d5hzm" event={"ID":"84fb351f-ede4-4f1d-a3ee-84565973a447","Type":"ContainerDied","Data":"2c65b76a5f73c2525728af0f971698db1348ad512360d62f13c173e76097ebeb"}
Oct 30 18:45:56.061516 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:56.061171    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-d5hzm" event={"ID":"84fb351f-ede4-4f1d-a3ee-84565973a447","Type":"ContainerDied","Data":"8d5faad3a902f941413f5c4ef6e391f66e2ea197577df0792b3f1e479bc9fd04"}
Oct 30 18:45:56.061516 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:56.061211    4924 scope.go:122] "RemoveContainer" containerID="2c65b76a5f73c2525728af0f971698db1348ad512360d62f13c173e76097ebeb"
Oct 30 18:45:56.070046 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:56.070032    4924 scope.go:122] "RemoveContainer" containerID="2c65b76a5f73c2525728af0f971698db1348ad512360d62f13c173e76097ebeb"
Oct 30 18:45:56.070321 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:45:56.070303    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c65b76a5f73c2525728af0f971698db1348ad512360d62f13c173e76097ebeb\": container with ID starting with 2c65b76a5f73c2525728af0f971698db1348ad512360d62f13c173e76097ebeb not found: ID does not exist" containerID="2c65b76a5f73c2525728af0f971698db1348ad512360d62f13c173e76097ebeb"
Oct 30 18:45:56.070368 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:56.070330    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c65b76a5f73c2525728af0f971698db1348ad512360d62f13c173e76097ebeb"} err="failed to get container status \"2c65b76a5f73c2525728af0f971698db1348ad512360d62f13c173e76097ebeb\": rpc error: code = NotFound desc = could not find container \"2c65b76a5f73c2525728af0f971698db1348ad512360d62f13c173e76097ebeb\": container with ID starting with 2c65b76a5f73c2525728af0f971698db1348ad512360d62f13c173e76097ebeb not found: ID does not exist"
Oct 30 18:45:56.216926 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:56.216896    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-d5hzm"]
Oct 30 18:45:56.263609 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:56.263582    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-d5hzm"]
Oct 30 18:45:56.832213 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:45:56.832182    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84fb351f-ede4-4f1d-a3ee-84565973a447" path="/var/lib/kubelet/pods/84fb351f-ede4-4f1d-a3ee-84565973a447/volumes"
Oct 30 18:46:11.809876 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:11.809836    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:12.031249 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:12.031217    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:12.633017 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:12.632984    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:15.709930 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:46:15.709900    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:46:25.061849 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:25.061768    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:27.347203 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:27.347164    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:27.611681 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:27.611596    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:27.653344 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:27.653307    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-gdb8r"]
Oct 30 18:46:27.653666 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:27.653651    4924 cpu_manager.go:414] "containerMap: removing container" podUID="84fb351f-ede4-4f1d-a3ee-84565973a447" containerName="authorino"
Oct 30 18:46:27.653711 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:27.653668    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84fb351f-ede4-4f1d-a3ee-84565973a447" containerName="authorino"
Oct 30 18:46:27.653745 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:27.653734    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="84fb351f-ede4-4f1d-a3ee-84565973a447" containerName="authorino"
Oct 30 18:46:27.657828 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:27.657807    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-gdb8r"
Oct 30 18:46:27.715146 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:27.715117    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-gdb8r"]
Oct 30 18:46:27.721216 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:27.721193    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x7ld\" (UniqueName: \"kubernetes.io/projected/e1f70ae2-9613-48d6-b643-7b6754e5fdb5-kube-api-access-8x7ld\") pod \"authorino-6f6944bcdd-gdb8r\" (UID: \"e1f70ae2-9613-48d6-b643-7b6754e5fdb5\") " pod="kuadrant-system/authorino-6f6944bcdd-gdb8r"
Oct 30 18:46:27.822158 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:27.822123    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x7ld\" (UniqueName: \"kubernetes.io/projected/e1f70ae2-9613-48d6-b643-7b6754e5fdb5-kube-api-access-8x7ld\") pod \"authorino-6f6944bcdd-gdb8r\" (UID: \"e1f70ae2-9613-48d6-b643-7b6754e5fdb5\") " pod="kuadrant-system/authorino-6f6944bcdd-gdb8r"
Oct 30 18:46:27.877891 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:27.877824    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x7ld\" (UniqueName: \"kubernetes.io/projected/e1f70ae2-9613-48d6-b643-7b6754e5fdb5-kube-api-access-8x7ld\") pod \"authorino-6f6944bcdd-gdb8r\" (UID: \"e1f70ae2-9613-48d6-b643-7b6754e5fdb5\") " pod="kuadrant-system/authorino-6f6944bcdd-gdb8r"
Oct 30 18:46:27.967679 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:27.967653    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-gdb8r"
Oct 30 18:46:28.026952 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:28.026921    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-gdb8r"]
Oct 30 18:46:28.052636 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:28.052610    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-587d4b867-gjjxn"]
Oct 30 18:46:28.055295 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:28.055271    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-587d4b867-gjjxn"
Oct 30 18:46:28.113202 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:28.113178    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-587d4b867-gjjxn"]
Oct 30 18:46:28.124490 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:28.124461    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb8lq\" (UniqueName: \"kubernetes.io/projected/e8d2973a-a724-4588-9a52-7b5aa5101bcd-kube-api-access-zb8lq\") pod \"authorino-587d4b867-gjjxn\" (UID: \"e8d2973a-a724-4588-9a52-7b5aa5101bcd\") " pod="kuadrant-system/authorino-587d4b867-gjjxn"
Oct 30 18:46:28.225683 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:28.225606    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb8lq\" (UniqueName: \"kubernetes.io/projected/e8d2973a-a724-4588-9a52-7b5aa5101bcd-kube-api-access-zb8lq\") pod \"authorino-587d4b867-gjjxn\" (UID: \"e8d2973a-a724-4588-9a52-7b5aa5101bcd\") " pod="kuadrant-system/authorino-587d4b867-gjjxn"
Oct 30 18:46:28.281628 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:28.281595    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb8lq\" (UniqueName: \"kubernetes.io/projected/e8d2973a-a724-4588-9a52-7b5aa5101bcd-kube-api-access-zb8lq\") pod \"authorino-587d4b867-gjjxn\" (UID: \"e8d2973a-a724-4588-9a52-7b5aa5101bcd\") " pod="kuadrant-system/authorino-587d4b867-gjjxn"
Oct 30 18:46:28.334664 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:28.334635    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-gdb8r"]
Oct 30 18:46:28.338038 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:46:28.338010    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1f70ae2_9613_48d6_b643_7b6754e5fdb5.slice/crio-9435c6cca43986aff8ec19c00766d6c8616dde9345dd35cc3689e46ad5473752 WatchSource:0}: Error finding container 9435c6cca43986aff8ec19c00766d6c8616dde9345dd35cc3689e46ad5473752: Status 404 returned error can't find the container with id 9435c6cca43986aff8ec19c00766d6c8616dde9345dd35cc3689e46ad5473752
Oct 30 18:46:28.366115 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:28.366089    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-587d4b867-gjjxn"
Oct 30 18:46:28.464431 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:28.464399    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-587d4b867-gjjxn"]
Oct 30 18:46:28.496641 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:28.496579    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f955ff69c-rmvbv"]
Oct 30 18:46:28.499439 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:28.499424    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f955ff69c-rmvbv"
Oct 30 18:46:28.528444 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:28.528423    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85kcl\" (UniqueName: \"kubernetes.io/projected/d09b131f-77e0-4e2b-a529-d7e0f5ab33a8-kube-api-access-85kcl\") pod \"authorino-5f955ff69c-rmvbv\" (UID: \"d09b131f-77e0-4e2b-a529-d7e0f5ab33a8\") " pod="kuadrant-system/authorino-5f955ff69c-rmvbv"
Oct 30 18:46:28.528528 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:28.528468    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d09b131f-77e0-4e2b-a529-d7e0f5ab33a8-tls-cert\") pod \"authorino-5f955ff69c-rmvbv\" (UID: \"d09b131f-77e0-4e2b-a529-d7e0f5ab33a8\") " pod="kuadrant-system/authorino-5f955ff69c-rmvbv"
Oct 30 18:46:28.552431 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:28.552412    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 18:46:28.556266 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:28.556243    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f955ff69c-rmvbv"]
Oct 30 18:46:28.629417 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:28.629393    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d09b131f-77e0-4e2b-a529-d7e0f5ab33a8-tls-cert\") pod \"authorino-5f955ff69c-rmvbv\" (UID: \"d09b131f-77e0-4e2b-a529-d7e0f5ab33a8\") " pod="kuadrant-system/authorino-5f955ff69c-rmvbv"
Oct 30 18:46:28.629543 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:28.629460    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85kcl\" (UniqueName: \"kubernetes.io/projected/d09b131f-77e0-4e2b-a529-d7e0f5ab33a8-kube-api-access-85kcl\") pod \"authorino-5f955ff69c-rmvbv\" (UID: \"d09b131f-77e0-4e2b-a529-d7e0f5ab33a8\") " pod="kuadrant-system/authorino-5f955ff69c-rmvbv"
Oct 30 18:46:28.631722 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:28.631705    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d09b131f-77e0-4e2b-a529-d7e0f5ab33a8-tls-cert\") pod \"authorino-5f955ff69c-rmvbv\" (UID: \"d09b131f-77e0-4e2b-a529-d7e0f5ab33a8\") " pod="kuadrant-system/authorino-5f955ff69c-rmvbv"
Oct 30 18:46:28.684249 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:28.684225    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85kcl\" (UniqueName: \"kubernetes.io/projected/d09b131f-77e0-4e2b-a529-d7e0f5ab33a8-kube-api-access-85kcl\") pod \"authorino-5f955ff69c-rmvbv\" (UID: \"d09b131f-77e0-4e2b-a529-d7e0f5ab33a8\") " pod="kuadrant-system/authorino-5f955ff69c-rmvbv"
Oct 30 18:46:28.725671 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:28.725649    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-587d4b867-gjjxn"]
Oct 30 18:46:28.810046 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:28.809989    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f955ff69c-rmvbv"
Oct 30 18:46:29.171698 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:29.171668    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f955ff69c-rmvbv"]
Oct 30 18:46:29.174386 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:46:29.174355    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd09b131f_77e0_4e2b_a529_d7e0f5ab33a8.slice/crio-e7cc9e0fbd862525081c6a9d82ee92011274fb6053720d766d146fbcbcc1afd5 WatchSource:0}: Error finding container e7cc9e0fbd862525081c6a9d82ee92011274fb6053720d766d146fbcbcc1afd5: Status 404 returned error can't find the container with id e7cc9e0fbd862525081c6a9d82ee92011274fb6053720d766d146fbcbcc1afd5
Oct 30 18:46:29.195554 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:29.195523    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-gdb8r" event={"ID":"e1f70ae2-9613-48d6-b643-7b6754e5fdb5","Type":"ContainerStarted","Data":"9435c6cca43986aff8ec19c00766d6c8616dde9345dd35cc3689e46ad5473752"}
Oct 30 18:46:29.196638 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:29.196616    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-587d4b867-gjjxn" event={"ID":"e8d2973a-a724-4588-9a52-7b5aa5101bcd","Type":"ContainerStarted","Data":"8b569016ee8ec69d1b03af4fa442ca0a8878c03b5b49cff0ef191addb0c833b0"}
Oct 30 18:46:29.197638 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:29.197616    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f955ff69c-rmvbv" event={"ID":"d09b131f-77e0-4e2b-a529-d7e0f5ab33a8","Type":"ContainerStarted","Data":"e7cc9e0fbd862525081c6a9d82ee92011274fb6053720d766d146fbcbcc1afd5"}
Oct 30 18:46:30.204351 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:30.204314    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-gdb8r" event={"ID":"e1f70ae2-9613-48d6-b643-7b6754e5fdb5","Type":"ContainerStarted","Data":"f3f4febae87ddb79741c46c78112936be20bdcd9c3251837092d5dc5b44d52cc"}
Oct 30 18:46:30.204804 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:30.204358    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-gdb8r" podUID="e1f70ae2-9613-48d6-b643-7b6754e5fdb5" containerName="authorino" containerID="cri-o://f3f4febae87ddb79741c46c78112936be20bdcd9c3251837092d5dc5b44d52cc" gracePeriod=30
Oct 30 18:46:30.205914 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:30.205893    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-587d4b867-gjjxn" event={"ID":"e8d2973a-a724-4588-9a52-7b5aa5101bcd","Type":"ContainerStarted","Data":"9fc3ba4fca5bf910ff1b28b36950dde6833c97b93bfecbce6ae693e467ee7a93"}
Oct 30 18:46:30.206003 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:30.205954    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-587d4b867-gjjxn" podUID="e8d2973a-a724-4588-9a52-7b5aa5101bcd" containerName="authorino" containerID="cri-o://9fc3ba4fca5bf910ff1b28b36950dde6833c97b93bfecbce6ae693e467ee7a93" gracePeriod=30
Oct 30 18:46:30.315809 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:30.315750    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-gdb8r" podStartSLOduration=1.91989677 podStartE2EDuration="3.315736643s" totalImagesPullingTime="1.395839873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:30.313634398 +0000 UTC m=+591.590684844" watchObservedRunningTime="2026-10-30 18:46:30.315736643 +0000 UTC m=+591.592787088"
Oct 30 18:46:30.429756 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:30.429706    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-587d4b867-gjjxn" podStartSLOduration=1.178394748 podStartE2EDuration="2.429693559s" totalImagesPullingTime="1.251298811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:30.427708199 +0000 UTC m=+591.704758645" watchObservedRunningTime="2026-10-30 18:46:30.429693559 +0000 UTC m=+591.706744004"
Oct 30 18:46:30.480683 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:30.480659    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-gdb8r"
Oct 30 18:46:30.498326 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:30.498304    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-587d4b867-gjjxn"
Oct 30 18:46:30.543000 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:30.542951    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8d2973a-a724-4588-9a52-7b5aa5101bcd-kube-api-access-zb8lq\" (UniqueName: \"kubernetes.io/projected/e8d2973a-a724-4588-9a52-7b5aa5101bcd-kube-api-access-zb8lq\") pod \"e8d2973a-a724-4588-9a52-7b5aa5101bcd\" (UID: \"e8d2973a-a724-4588-9a52-7b5aa5101bcd\") "
Oct 30 18:46:30.543123 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:30.543010    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1f70ae2-9613-48d6-b643-7b6754e5fdb5-kube-api-access-8x7ld\" (UniqueName: \"kubernetes.io/projected/e1f70ae2-9613-48d6-b643-7b6754e5fdb5-kube-api-access-8x7ld\") pod \"e1f70ae2-9613-48d6-b643-7b6754e5fdb5\" (UID: \"e1f70ae2-9613-48d6-b643-7b6754e5fdb5\") "
Oct 30 18:46:30.544781 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:30.544759    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f70ae2-9613-48d6-b643-7b6754e5fdb5-kube-api-access-8x7ld" pod "e1f70ae2-9613-48d6-b643-7b6754e5fdb5" (UID: "e1f70ae2-9613-48d6-b643-7b6754e5fdb5"). InnerVolumeSpecName "kube-api-access-8x7ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:46:30.544886 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:30.544860    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d2973a-a724-4588-9a52-7b5aa5101bcd-kube-api-access-zb8lq" pod "e8d2973a-a724-4588-9a52-7b5aa5101bcd" (UID: "e8d2973a-a724-4588-9a52-7b5aa5101bcd"). InnerVolumeSpecName "kube-api-access-zb8lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:46:30.644058 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:30.644023    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x7ld\" (UniqueName: \"kubernetes.io/projected/e1f70ae2-9613-48d6-b643-7b6754e5fdb5-kube-api-access-8x7ld\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:46:30.644058 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:30.644053    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb8lq\" (UniqueName: \"kubernetes.io/projected/e8d2973a-a724-4588-9a52-7b5aa5101bcd-kube-api-access-zb8lq\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:46:31.212055 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.212014    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f955ff69c-rmvbv" event={"ID":"d09b131f-77e0-4e2b-a529-d7e0f5ab33a8","Type":"ContainerStarted","Data":"6c4190b4b4cc568ea4c73a2837aa7a443fde4a93e6acbb974fa55aaa5ce1138f"}
Oct 30 18:46:31.213254 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.213228    4924 generic.go:356] "Generic (PLEG): container finished" podID="e1f70ae2-9613-48d6-b643-7b6754e5fdb5" containerID="f3f4febae87ddb79741c46c78112936be20bdcd9c3251837092d5dc5b44d52cc" exitCode=0
Oct 30 18:46:31.213387 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.213262    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-gdb8r"
Oct 30 18:46:31.213387 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.213314    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-gdb8r" event={"ID":"e1f70ae2-9613-48d6-b643-7b6754e5fdb5","Type":"ContainerDied","Data":"f3f4febae87ddb79741c46c78112936be20bdcd9c3251837092d5dc5b44d52cc"}
Oct 30 18:46:31.213387 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.213358    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-gdb8r" event={"ID":"e1f70ae2-9613-48d6-b643-7b6754e5fdb5","Type":"ContainerDied","Data":"9435c6cca43986aff8ec19c00766d6c8616dde9345dd35cc3689e46ad5473752"}
Oct 30 18:46:31.213387 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.213379    4924 scope.go:122] "RemoveContainer" containerID="f3f4febae87ddb79741c46c78112936be20bdcd9c3251837092d5dc5b44d52cc"
Oct 30 18:46:31.214752 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.214729    4924 generic.go:356] "Generic (PLEG): container finished" podID="e8d2973a-a724-4588-9a52-7b5aa5101bcd" containerID="9fc3ba4fca5bf910ff1b28b36950dde6833c97b93bfecbce6ae693e467ee7a93" exitCode=0
Oct 30 18:46:31.214834 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.214759    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-587d4b867-gjjxn"
Oct 30 18:46:31.214834 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.214783    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-587d4b867-gjjxn" event={"ID":"e8d2973a-a724-4588-9a52-7b5aa5101bcd","Type":"ContainerDied","Data":"9fc3ba4fca5bf910ff1b28b36950dde6833c97b93bfecbce6ae693e467ee7a93"}
Oct 30 18:46:31.214834 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.214828    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-587d4b867-gjjxn" event={"ID":"e8d2973a-a724-4588-9a52-7b5aa5101bcd","Type":"ContainerDied","Data":"8b569016ee8ec69d1b03af4fa442ca0a8878c03b5b49cff0ef191addb0c833b0"}
Oct 30 18:46:31.221283 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.221267    4924 scope.go:122] "RemoveContainer" containerID="f3f4febae87ddb79741c46c78112936be20bdcd9c3251837092d5dc5b44d52cc"
Oct 30 18:46:31.221538 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:46:31.221519    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3f4febae87ddb79741c46c78112936be20bdcd9c3251837092d5dc5b44d52cc\": container with ID starting with f3f4febae87ddb79741c46c78112936be20bdcd9c3251837092d5dc5b44d52cc not found: ID does not exist" containerID="f3f4febae87ddb79741c46c78112936be20bdcd9c3251837092d5dc5b44d52cc"
Oct 30 18:46:31.221609 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.221549    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3f4febae87ddb79741c46c78112936be20bdcd9c3251837092d5dc5b44d52cc"} err="failed to get container status \"f3f4febae87ddb79741c46c78112936be20bdcd9c3251837092d5dc5b44d52cc\": rpc error: code = NotFound desc = could not find container \"f3f4febae87ddb79741c46c78112936be20bdcd9c3251837092d5dc5b44d52cc\": container with ID starting with f3f4febae87ddb79741c46c78112936be20bdcd9c3251837092d5dc5b44d52cc not found: ID does not exist"
Oct 30 18:46:31.221609 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.221570    4924 scope.go:122] "RemoveContainer" containerID="9fc3ba4fca5bf910ff1b28b36950dde6833c97b93bfecbce6ae693e467ee7a93"
Oct 30 18:46:31.228619 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.228603    4924 scope.go:122] "RemoveContainer" containerID="9fc3ba4fca5bf910ff1b28b36950dde6833c97b93bfecbce6ae693e467ee7a93"
Oct 30 18:46:31.228847 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:46:31.228828    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fc3ba4fca5bf910ff1b28b36950dde6833c97b93bfecbce6ae693e467ee7a93\": container with ID starting with 9fc3ba4fca5bf910ff1b28b36950dde6833c97b93bfecbce6ae693e467ee7a93 not found: ID does not exist" containerID="9fc3ba4fca5bf910ff1b28b36950dde6833c97b93bfecbce6ae693e467ee7a93"
Oct 30 18:46:31.228904 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.228854    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fc3ba4fca5bf910ff1b28b36950dde6833c97b93bfecbce6ae693e467ee7a93"} err="failed to get container status \"9fc3ba4fca5bf910ff1b28b36950dde6833c97b93bfecbce6ae693e467ee7a93\": rpc error: code = NotFound desc = could not find container \"9fc3ba4fca5bf910ff1b28b36950dde6833c97b93bfecbce6ae693e467ee7a93\": container with ID starting with 9fc3ba4fca5bf910ff1b28b36950dde6833c97b93bfecbce6ae693e467ee7a93 not found: ID does not exist"
Oct 30 18:46:31.324150 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.324102    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5f955ff69c-rmvbv" podStartSLOduration=2.04801488 podStartE2EDuration="3.324088667s" totalImagesPullingTime="1.276073787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:31.323887934 +0000 UTC m=+592.600938379" watchObservedRunningTime="2026-10-30 18:46:31.324088667 +0000 UTC m=+592.601139132"
Oct 30 18:46:31.357762 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.357728    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-b8gz8"]
Oct 30 18:46:31.357940 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.357923    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-b8gz8" podUID="9908a9c0-7387-4e15-bbc0-6d58d7c3fb72" containerName="authorino" containerID="cri-o://f9f2f54a7f4b9151fce2ea937a0cc3efc7fbb75ac96d0165278e7d4363b12c57" gracePeriod=30
Oct 30 18:46:31.429947 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.429915    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-587d4b867-gjjxn"]
Oct 30 18:46:31.434288 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.434263    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-587d4b867-gjjxn"]
Oct 30 18:46:31.549175 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.549147    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-gdb8r"]
Oct 30 18:46:31.551814 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.551787    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-gdb8r"]
Oct 30 18:46:31.572402 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.572374    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:31.610954 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.610932    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-b8gz8"
Oct 30 18:46:31.652897 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.652866    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9908a9c0-7387-4e15-bbc0-6d58d7c3fb72-kube-api-access-lqnvt\" (UniqueName: \"kubernetes.io/projected/9908a9c0-7387-4e15-bbc0-6d58d7c3fb72-kube-api-access-lqnvt\") pod \"9908a9c0-7387-4e15-bbc0-6d58d7c3fb72\" (UID: \"9908a9c0-7387-4e15-bbc0-6d58d7c3fb72\") "
Oct 30 18:46:31.654858 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.654835    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9908a9c0-7387-4e15-bbc0-6d58d7c3fb72-kube-api-access-lqnvt" pod "9908a9c0-7387-4e15-bbc0-6d58d7c3fb72" (UID: "9908a9c0-7387-4e15-bbc0-6d58d7c3fb72"). InnerVolumeSpecName "kube-api-access-lqnvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:46:31.754180 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:31.754109    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqnvt\" (UniqueName: \"kubernetes.io/projected/9908a9c0-7387-4e15-bbc0-6d58d7c3fb72-kube-api-access-lqnvt\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:46:32.221465 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:32.221433    4924 generic.go:356] "Generic (PLEG): container finished" podID="9908a9c0-7387-4e15-bbc0-6d58d7c3fb72" containerID="f9f2f54a7f4b9151fce2ea937a0cc3efc7fbb75ac96d0165278e7d4363b12c57" exitCode=0
Oct 30 18:46:32.221932 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:32.221478    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-b8gz8"
Oct 30 18:46:32.221932 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:32.221511    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-b8gz8" event={"ID":"9908a9c0-7387-4e15-bbc0-6d58d7c3fb72","Type":"ContainerDied","Data":"f9f2f54a7f4b9151fce2ea937a0cc3efc7fbb75ac96d0165278e7d4363b12c57"}
Oct 30 18:46:32.221932 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:32.221545    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-b8gz8" event={"ID":"9908a9c0-7387-4e15-bbc0-6d58d7c3fb72","Type":"ContainerDied","Data":"614bea46a1c7ccfe341f9518427c12d746984ab78aa2294b02cd106bc54ef89d"}
Oct 30 18:46:32.221932 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:32.221566    4924 scope.go:122] "RemoveContainer" containerID="f9f2f54a7f4b9151fce2ea937a0cc3efc7fbb75ac96d0165278e7d4363b12c57"
Oct 30 18:46:32.229742 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:32.229722    4924 scope.go:122] "RemoveContainer" containerID="f9f2f54a7f4b9151fce2ea937a0cc3efc7fbb75ac96d0165278e7d4363b12c57"
Oct 30 18:46:32.230023 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:46:32.229997    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9f2f54a7f4b9151fce2ea937a0cc3efc7fbb75ac96d0165278e7d4363b12c57\": container with ID starting with f9f2f54a7f4b9151fce2ea937a0cc3efc7fbb75ac96d0165278e7d4363b12c57 not found: ID does not exist" containerID="f9f2f54a7f4b9151fce2ea937a0cc3efc7fbb75ac96d0165278e7d4363b12c57"
Oct 30 18:46:32.230135 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:32.230029    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f2f54a7f4b9151fce2ea937a0cc3efc7fbb75ac96d0165278e7d4363b12c57"} err="failed to get container status \"f9f2f54a7f4b9151fce2ea937a0cc3efc7fbb75ac96d0165278e7d4363b12c57\": rpc error: code = NotFound desc = could not find container \"f9f2f54a7f4b9151fce2ea937a0cc3efc7fbb75ac96d0165278e7d4363b12c57\": container with ID starting with f9f2f54a7f4b9151fce2ea937a0cc3efc7fbb75ac96d0165278e7d4363b12c57 not found: ID does not exist"
Oct 30 18:46:32.379967 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:32.379934    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-b8gz8"]
Oct 30 18:46:32.386405 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:32.386382    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-b8gz8"]
Oct 30 18:46:32.833800 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:32.833770    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9908a9c0-7387-4e15-bbc0-6d58d7c3fb72" path="/var/lib/kubelet/pods/9908a9c0-7387-4e15-bbc0-6d58d7c3fb72/volumes"
Oct 30 18:46:32.834109 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:32.834096    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1f70ae2-9613-48d6-b643-7b6754e5fdb5" path="/var/lib/kubelet/pods/e1f70ae2-9613-48d6-b643-7b6754e5fdb5/volumes"
Oct 30 18:46:32.834374 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:32.834363    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8d2973a-a724-4588-9a52-7b5aa5101bcd" path="/var/lib/kubelet/pods/e8d2973a-a724-4588-9a52-7b5aa5101bcd/volumes"
Oct 30 18:46:38.777300 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:38.777259    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:38.873458 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:38.873434    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 18:46:38.875153 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:38.875134    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 18:46:38.965820 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:38.965780    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:39.207845 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:39.207815    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:39.389510 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:39.389478    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:45.444458 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:45.444422    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:45.635058 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:45.635027    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:45.710551 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:46:45.710472    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:46:50.297520 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:50.297476    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:50.478088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:50.478027    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:57.851905 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:57.851871    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:46:58.041636 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:46:58.041607    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:47:15.710290 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:47:15.710256    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:47:18.848319 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:47:18.848277    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:47:19.092371 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:47:19.092342    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:47:20.792786 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:47:20.792748    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:47:20.984282 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:47:20.984248    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:47:21.226446 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:47:21.226416    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:47:28.038192 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:47:28.038157    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:47:28.229603 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:47:28.229547    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:47:31.894958 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:47:31.894925    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:47:45.710291 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:47:45.710253    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:48:09.273172 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:09.273082    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:48:09.462102 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:09.462047    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:48:15.709852 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:48:15.709816    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:48:28.780117 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.780081    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"]
Oct 30 18:48:28.780541 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.780424    4924 cpu_manager.go:414] "containerMap: removing container" podUID="e8d2973a-a724-4588-9a52-7b5aa5101bcd" containerName="authorino"
Oct 30 18:48:28.780541 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.780439    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8d2973a-a724-4588-9a52-7b5aa5101bcd" containerName="authorino"
Oct 30 18:48:28.780541 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.780466    4924 cpu_manager.go:414] "containerMap: removing container" podUID="9908a9c0-7387-4e15-bbc0-6d58d7c3fb72" containerName="authorino"
Oct 30 18:48:28.780541 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.780475    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9908a9c0-7387-4e15-bbc0-6d58d7c3fb72" containerName="authorino"
Oct 30 18:48:28.780541 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.780496    4924 cpu_manager.go:414] "containerMap: removing container" podUID="e1f70ae2-9613-48d6-b643-7b6754e5fdb5" containerName="authorino"
Oct 30 18:48:28.780541 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.780503    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1f70ae2-9613-48d6-b643-7b6754e5fdb5" containerName="authorino"
Oct 30 18:48:28.780784 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.780597    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8d2973a-a724-4588-9a52-7b5aa5101bcd" containerName="authorino"
Oct 30 18:48:28.780784 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.780607    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="9908a9c0-7387-4e15-bbc0-6d58d7c3fb72" containerName="authorino"
Oct 30 18:48:28.780784 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.780614    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1f70ae2-9613-48d6-b643-7b6754e5fdb5" containerName="authorino"
Oct 30 18:48:28.783468 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.783449    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"
Oct 30 18:48:28.833584 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.833341    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 18:48:28.879433 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.879411    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 18:48:28.879589 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.879575    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 18:48:28.879673 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.879660    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7nxf2\""
Oct 30 18:48:28.881112 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.881094    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"]
Oct 30 18:48:28.890354 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.890337    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d49737a-ecba-4943-a8c5-419f1a1f115a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs\" (UID: \"4d49737a-ecba-4943-a8c5-419f1a1f115a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"
Oct 30 18:48:28.890409 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.890364    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d49737a-ecba-4943-a8c5-419f1a1f115a-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs\" (UID: \"4d49737a-ecba-4943-a8c5-419f1a1f115a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"
Oct 30 18:48:28.890577 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.890556    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d49737a-ecba-4943-a8c5-419f1a1f115a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs\" (UID: \"4d49737a-ecba-4943-a8c5-419f1a1f115a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"
Oct 30 18:48:28.890707 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.890688    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d49737a-ecba-4943-a8c5-419f1a1f115a-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs\" (UID: \"4d49737a-ecba-4943-a8c5-419f1a1f115a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"
Oct 30 18:48:28.890750 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.890716    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc6g4\" (UniqueName: \"kubernetes.io/projected/4d49737a-ecba-4943-a8c5-419f1a1f115a-kube-api-access-kc6g4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs\" (UID: \"4d49737a-ecba-4943-a8c5-419f1a1f115a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"
Oct 30 18:48:28.890825 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.890812    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d49737a-ecba-4943-a8c5-419f1a1f115a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs\" (UID: \"4d49737a-ecba-4943-a8c5-419f1a1f115a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"
Oct 30 18:48:28.936790 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.936766    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:48:28.991410 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.991387    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d49737a-ecba-4943-a8c5-419f1a1f115a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs\" (UID: \"4d49737a-ecba-4943-a8c5-419f1a1f115a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"
Oct 30 18:48:28.991517 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.991426    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d49737a-ecba-4943-a8c5-419f1a1f115a-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs\" (UID: \"4d49737a-ecba-4943-a8c5-419f1a1f115a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"
Oct 30 18:48:28.991517 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.991443    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc6g4\" (UniqueName: \"kubernetes.io/projected/4d49737a-ecba-4943-a8c5-419f1a1f115a-kube-api-access-kc6g4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs\" (UID: \"4d49737a-ecba-4943-a8c5-419f1a1f115a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"
Oct 30 18:48:28.991517 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.991475    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d49737a-ecba-4943-a8c5-419f1a1f115a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs\" (UID: \"4d49737a-ecba-4943-a8c5-419f1a1f115a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"
Oct 30 18:48:28.991517 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.991513    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d49737a-ecba-4943-a8c5-419f1a1f115a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs\" (UID: \"4d49737a-ecba-4943-a8c5-419f1a1f115a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"
Oct 30 18:48:28.991669 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.991529    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d49737a-ecba-4943-a8c5-419f1a1f115a-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs\" (UID: \"4d49737a-ecba-4943-a8c5-419f1a1f115a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"
Oct 30 18:48:28.991847 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.991824    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d49737a-ecba-4943-a8c5-419f1a1f115a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs\" (UID: \"4d49737a-ecba-4943-a8c5-419f1a1f115a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"
Oct 30 18:48:28.991908 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.991866    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d49737a-ecba-4943-a8c5-419f1a1f115a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs\" (UID: \"4d49737a-ecba-4943-a8c5-419f1a1f115a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"
Oct 30 18:48:28.991964 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.991914    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d49737a-ecba-4943-a8c5-419f1a1f115a-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs\" (UID: \"4d49737a-ecba-4943-a8c5-419f1a1f115a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"
Oct 30 18:48:28.993807 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.993783    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d49737a-ecba-4943-a8c5-419f1a1f115a-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs\" (UID: \"4d49737a-ecba-4943-a8c5-419f1a1f115a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"
Oct 30 18:48:28.993996 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:28.993983    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d49737a-ecba-4943-a8c5-419f1a1f115a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs\" (UID: \"4d49737a-ecba-4943-a8c5-419f1a1f115a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"
Oct 30 18:48:29.045523 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:29.045472    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc6g4\" (UniqueName: \"kubernetes.io/projected/4d49737a-ecba-4943-a8c5-419f1a1f115a-kube-api-access-kc6g4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs\" (UID: \"4d49737a-ecba-4943-a8c5-419f1a1f115a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"
Oct 30 18:48:29.095220 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:29.095196    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"
Oct 30 18:48:29.458427 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:29.458306    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"]
Oct 30 18:48:29.461016 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:48:29.460992    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d49737a_ecba_4943_a8c5_419f1a1f115a.slice/crio-3a3d3c489bbe4197d1004fe56758fa92bb75c2ea92d6423cd25f801faa8ecc15 WatchSource:0}: Error finding container 3a3d3c489bbe4197d1004fe56758fa92bb75c2ea92d6423cd25f801faa8ecc15: Status 404 returned error can't find the container with id 3a3d3c489bbe4197d1004fe56758fa92bb75c2ea92d6423cd25f801faa8ecc15
Oct 30 18:48:29.462542 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:29.462522    4924 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:48:29.696677 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:29.696644    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs" event={"ID":"4d49737a-ecba-4943-a8c5-419f1a1f115a","Type":"ContainerStarted","Data":"3a3d3c489bbe4197d1004fe56758fa92bb75c2ea92d6423cd25f801faa8ecc15"}
Oct 30 18:48:31.706907 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:31.706869    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs" event={"ID":"4d49737a-ecba-4943-a8c5-419f1a1f115a","Type":"ContainerStarted","Data":"12a35731526ff9ef931ff537d4f6cd839d1ea16d65a1c10b4dda625f8427e22d"}
Oct 30 18:48:31.816372 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:31.816323    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs" podStartSLOduration=1.706410371 podStartE2EDuration="3.816310145s" totalImagesPullingTime="2.109899774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:31.815864917 +0000 UTC m=+713.092915363" watchObservedRunningTime="2026-10-30 18:48:31.816310145 +0000 UTC m=+713.093360584"
Oct 30 18:48:32.096259 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:32.096224    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"
Oct 30 18:48:32.228194 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:32.228155    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:48:32.626970 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:32.626935    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"]
Oct 30 18:48:32.630380 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:32.630357    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:48:32.725747 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:32.725690    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 18:48:32.726966 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:32.726944    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3d792df-177d-4f7a-bba8-f6351175462e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd\" (UID: \"a3d792df-177d-4f7a-bba8-f6351175462e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:48:32.727091 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:32.727010    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3d792df-177d-4f7a-bba8-f6351175462e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd\" (UID: \"a3d792df-177d-4f7a-bba8-f6351175462e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:48:32.727091 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:32.727043    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swnfk\" (UniqueName: \"kubernetes.io/projected/a3d792df-177d-4f7a-bba8-f6351175462e-kube-api-access-swnfk\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd\" (UID: \"a3d792df-177d-4f7a-bba8-f6351175462e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:48:32.727189 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:32.727141    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"]
Oct 30 18:48:32.727189 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:32.727156    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3d792df-177d-4f7a-bba8-f6351175462e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd\" (UID: \"a3d792df-177d-4f7a-bba8-f6351175462e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:48:32.727189 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:32.727184    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3d792df-177d-4f7a-bba8-f6351175462e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd\" (UID: \"a3d792df-177d-4f7a-bba8-f6351175462e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:48:32.727299 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:32.727211    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3d792df-177d-4f7a-bba8-f6351175462e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd\" (UID: \"a3d792df-177d-4f7a-bba8-f6351175462e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:48:32.777922 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:32.777886    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:48:32.827705 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:32.827668    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3d792df-177d-4f7a-bba8-f6351175462e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd\" (UID: \"a3d792df-177d-4f7a-bba8-f6351175462e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:48:32.827906 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:32.827722    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3d792df-177d-4f7a-bba8-f6351175462e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd\" (UID: \"a3d792df-177d-4f7a-bba8-f6351175462e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:48:32.827906 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:32.827768    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3d792df-177d-4f7a-bba8-f6351175462e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd\" (UID: \"a3d792df-177d-4f7a-bba8-f6351175462e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:48:32.828031 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:32.827975    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3d792df-177d-4f7a-bba8-f6351175462e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd\" (UID: \"a3d792df-177d-4f7a-bba8-f6351175462e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:48:32.828031 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:32.828016    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3d792df-177d-4f7a-bba8-f6351175462e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd\" (UID: \"a3d792df-177d-4f7a-bba8-f6351175462e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:48:32.828180 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:32.828052    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swnfk\" (UniqueName: \"kubernetes.io/projected/a3d792df-177d-4f7a-bba8-f6351175462e-kube-api-access-swnfk\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd\" (UID: \"a3d792df-177d-4f7a-bba8-f6351175462e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:48:32.828180 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:32.828161    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3d792df-177d-4f7a-bba8-f6351175462e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd\" (UID: \"a3d792df-177d-4f7a-bba8-f6351175462e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:48:32.828287 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:32.828191    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3d792df-177d-4f7a-bba8-f6351175462e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd\" (UID: \"a3d792df-177d-4f7a-bba8-f6351175462e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:48:32.828447 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:32.828426    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3d792df-177d-4f7a-bba8-f6351175462e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd\" (UID: \"a3d792df-177d-4f7a-bba8-f6351175462e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:48:32.830147 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:32.830125    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3d792df-177d-4f7a-bba8-f6351175462e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd\" (UID: \"a3d792df-177d-4f7a-bba8-f6351175462e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:48:32.830624 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:32.830600    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3d792df-177d-4f7a-bba8-f6351175462e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd\" (UID: \"a3d792df-177d-4f7a-bba8-f6351175462e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:48:32.883744 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:32.883659    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swnfk\" (UniqueName: \"kubernetes.io/projected/a3d792df-177d-4f7a-bba8-f6351175462e-kube-api-access-swnfk\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd\" (UID: \"a3d792df-177d-4f7a-bba8-f6351175462e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:48:32.943627 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:32.943595    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:48:33.302154 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:33.302130    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"]
Oct 30 18:48:33.304038 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:48:33.304009    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3d792df_177d_4f7a_bba8_f6351175462e.slice/crio-86a6797ceda8a4196b1e40181c8051f7b40bcb12121b7309ddd7acac0db2d0e1 WatchSource:0}: Error finding container 86a6797ceda8a4196b1e40181c8051f7b40bcb12121b7309ddd7acac0db2d0e1: Status 404 returned error can't find the container with id 86a6797ceda8a4196b1e40181c8051f7b40bcb12121b7309ddd7acac0db2d0e1
Oct 30 18:48:33.717566 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:33.717531    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd" event={"ID":"a3d792df-177d-4f7a-bba8-f6351175462e","Type":"ContainerStarted","Data":"86a6797ceda8a4196b1e40181c8051f7b40bcb12121b7309ddd7acac0db2d0e1"}
Oct 30 18:48:34.722503 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:34.722470    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd" event={"ID":"a3d792df-177d-4f7a-bba8-f6351175462e","Type":"ContainerStarted","Data":"1dcc5365cad503b096e7c28b4dfeb9c3ffcf0fb088250e621f411ef1216e7d74"}
Oct 30 18:48:34.832136 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:34.832056    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd" podStartSLOduration=2.24173278 podStartE2EDuration="2.832039246s" totalImagesPullingTime="590.306466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:34.830616869 +0000 UTC m=+716.107667315" watchObservedRunningTime="2026-10-30 18:48:34.832039246 +0000 UTC m=+716.109089692"
Oct 30 18:48:34.944457 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:34.944430    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:48:37.074876 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:37.074817    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:48:37.695290 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:37.695255    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:48:37.872190 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:37.872164    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:48:39.095835 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:39.095806    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"
Oct 30 18:48:39.108287 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:39.108267    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"
Oct 30 18:48:39.752990 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:39.752962    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs"
Oct 30 18:48:41.126058 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:41.126021    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:48:42.937736 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:42.937702    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:48:42.944660 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:42.944638    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:48:42.956831 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:42.956812    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:48:43.127027 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:43.126972    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:48:43.789764 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:43.789733    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:48:45.546966 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:45.546932    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"]
Oct 30 18:48:45.549975 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:45.549959    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"
Oct 30 18:48:45.635912 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:45.635884    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ef7cef8-578d-4ce6-8479-990c27e212cb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fd8zz\" (UID: \"3ef7cef8-578d-4ce6-8479-990c27e212cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"
Oct 30 18:48:45.635912 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:45.635915    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef7cef8-578d-4ce6-8479-990c27e212cb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fd8zz\" (UID: \"3ef7cef8-578d-4ce6-8479-990c27e212cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"
Oct 30 18:48:45.636197 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:45.635934    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ef7cef8-578d-4ce6-8479-990c27e212cb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fd8zz\" (UID: \"3ef7cef8-578d-4ce6-8479-990c27e212cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"
Oct 30 18:48:45.636197 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:45.635954    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ef7cef8-578d-4ce6-8479-990c27e212cb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fd8zz\" (UID: \"3ef7cef8-578d-4ce6-8479-990c27e212cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"
Oct 30 18:48:45.636197 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:45.636028    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfqwr\" (UniqueName: \"kubernetes.io/projected/3ef7cef8-578d-4ce6-8479-990c27e212cb-kube-api-access-dfqwr\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fd8zz\" (UID: \"3ef7cef8-578d-4ce6-8479-990c27e212cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"
Oct 30 18:48:45.636197 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:45.636116    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef7cef8-578d-4ce6-8479-990c27e212cb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fd8zz\" (UID: \"3ef7cef8-578d-4ce6-8479-990c27e212cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"
Oct 30 18:48:45.645356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:45.645327    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 30 18:48:45.646830 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:45.646815    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"]
Oct 30 18:48:45.698774 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:45.698747    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:48:45.709897 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:48:45.709868    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:48:45.737375 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:45.737356    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ef7cef8-578d-4ce6-8479-990c27e212cb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fd8zz\" (UID: \"3ef7cef8-578d-4ce6-8479-990c27e212cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"
Oct 30 18:48:45.737500 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:45.737384    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef7cef8-578d-4ce6-8479-990c27e212cb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fd8zz\" (UID: \"3ef7cef8-578d-4ce6-8479-990c27e212cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"
Oct 30 18:48:45.737500 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:45.737403    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ef7cef8-578d-4ce6-8479-990c27e212cb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fd8zz\" (UID: \"3ef7cef8-578d-4ce6-8479-990c27e212cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"
Oct 30 18:48:45.737500 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:45.737423    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ef7cef8-578d-4ce6-8479-990c27e212cb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fd8zz\" (UID: \"3ef7cef8-578d-4ce6-8479-990c27e212cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"
Oct 30 18:48:45.737500 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:45.737439    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfqwr\" (UniqueName: \"kubernetes.io/projected/3ef7cef8-578d-4ce6-8479-990c27e212cb-kube-api-access-dfqwr\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fd8zz\" (UID: \"3ef7cef8-578d-4ce6-8479-990c27e212cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"
Oct 30 18:48:45.737500 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:45.737469    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef7cef8-578d-4ce6-8479-990c27e212cb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fd8zz\" (UID: \"3ef7cef8-578d-4ce6-8479-990c27e212cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"
Oct 30 18:48:45.737805 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:45.737780    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ef7cef8-578d-4ce6-8479-990c27e212cb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fd8zz\" (UID: \"3ef7cef8-578d-4ce6-8479-990c27e212cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"
Oct 30 18:48:45.737910 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:45.737880    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ef7cef8-578d-4ce6-8479-990c27e212cb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fd8zz\" (UID: \"3ef7cef8-578d-4ce6-8479-990c27e212cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"
Oct 30 18:48:45.737910 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:45.737906    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef7cef8-578d-4ce6-8479-990c27e212cb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fd8zz\" (UID: \"3ef7cef8-578d-4ce6-8479-990c27e212cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"
Oct 30 18:48:45.739564 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:45.739541    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ef7cef8-578d-4ce6-8479-990c27e212cb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fd8zz\" (UID: \"3ef7cef8-578d-4ce6-8479-990c27e212cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"
Oct 30 18:48:45.739964 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:45.739944    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef7cef8-578d-4ce6-8479-990c27e212cb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fd8zz\" (UID: \"3ef7cef8-578d-4ce6-8479-990c27e212cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"
Oct 30 18:48:45.791974 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:45.791945    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfqwr\" (UniqueName: \"kubernetes.io/projected/3ef7cef8-578d-4ce6-8479-990c27e212cb-kube-api-access-dfqwr\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-fd8zz\" (UID: \"3ef7cef8-578d-4ce6-8479-990c27e212cb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"
Oct 30 18:48:45.861347 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:45.861320    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"
Oct 30 18:48:46.220947 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:46.220918    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"]
Oct 30 18:48:46.223295 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:48:46.223264    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef7cef8_578d_4ce6_8479_990c27e212cb.slice/crio-15241e96b7a831957d0214a16569702053b915fef166b4154d7a4ebbf6049cb7 WatchSource:0}: Error finding container 15241e96b7a831957d0214a16569702053b915fef166b4154d7a4ebbf6049cb7: Status 404 returned error can't find the container with id 15241e96b7a831957d0214a16569702053b915fef166b4154d7a4ebbf6049cb7
Oct 30 18:48:46.786657 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:46.786625    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz" event={"ID":"3ef7cef8-578d-4ce6-8479-990c27e212cb","Type":"ContainerStarted","Data":"15241e96b7a831957d0214a16569702053b915fef166b4154d7a4ebbf6049cb7"}
Oct 30 18:48:47.797845 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:47.797809    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz" event={"ID":"3ef7cef8-578d-4ce6-8479-990c27e212cb","Type":"ContainerStarted","Data":"35f89e28d830102f82d67fdc150c56f0403058bb8cd0f28a3f91140a0d3443a3"}
Oct 30 18:48:47.862406 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:47.862366    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"
Oct 30 18:48:47.907831 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:47.907774    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz" podStartSLOduration=2.329347649 podStartE2EDuration="2.907759738s" totalImagesPullingTime="578.412089ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:47.906748301 +0000 UTC m=+729.183798761" watchObservedRunningTime="2026-10-30 18:48:47.907759738 +0000 UTC m=+729.184810230"
Oct 30 18:48:49.813019 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:49.812988    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"]
Oct 30 18:48:49.816474 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:49.816444    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"
Oct 30 18:48:49.912044 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:49.912008    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 30 18:48:49.913451 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:49.913430    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"]
Oct 30 18:48:49.952183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:49.952152    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:48:49.976258 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:49.976233    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18bb436d-4071-4558-9428-9dfb7daebbe4-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z6dn4\" (UID: \"18bb436d-4071-4558-9428-9dfb7daebbe4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"
Oct 30 18:48:49.976367 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:49.976280    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18bb436d-4071-4558-9428-9dfb7daebbe4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z6dn4\" (UID: \"18bb436d-4071-4558-9428-9dfb7daebbe4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"
Oct 30 18:48:49.976367 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:49.976299    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18bb436d-4071-4558-9428-9dfb7daebbe4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z6dn4\" (UID: \"18bb436d-4071-4558-9428-9dfb7daebbe4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"
Oct 30 18:48:49.976488 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:49.976425    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18bb436d-4071-4558-9428-9dfb7daebbe4-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z6dn4\" (UID: \"18bb436d-4071-4558-9428-9dfb7daebbe4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"
Oct 30 18:48:49.976488 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:49.976460    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18bb436d-4071-4558-9428-9dfb7daebbe4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z6dn4\" (UID: \"18bb436d-4071-4558-9428-9dfb7daebbe4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"
Oct 30 18:48:49.976577 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:49.976516    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md2rf\" (UniqueName: \"kubernetes.io/projected/18bb436d-4071-4558-9428-9dfb7daebbe4-kube-api-access-md2rf\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z6dn4\" (UID: \"18bb436d-4071-4558-9428-9dfb7daebbe4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"
Oct 30 18:48:50.077824 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:50.077741    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18bb436d-4071-4558-9428-9dfb7daebbe4-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z6dn4\" (UID: \"18bb436d-4071-4558-9428-9dfb7daebbe4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"
Oct 30 18:48:50.077824 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:50.077793    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18bb436d-4071-4558-9428-9dfb7daebbe4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z6dn4\" (UID: \"18bb436d-4071-4558-9428-9dfb7daebbe4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"
Oct 30 18:48:50.077824 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:50.077821    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18bb436d-4071-4558-9428-9dfb7daebbe4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z6dn4\" (UID: \"18bb436d-4071-4558-9428-9dfb7daebbe4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"
Oct 30 18:48:50.078106 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:50.077848    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18bb436d-4071-4558-9428-9dfb7daebbe4-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z6dn4\" (UID: \"18bb436d-4071-4558-9428-9dfb7daebbe4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"
Oct 30 18:48:50.078106 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:50.077874    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18bb436d-4071-4558-9428-9dfb7daebbe4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z6dn4\" (UID: \"18bb436d-4071-4558-9428-9dfb7daebbe4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"
Oct 30 18:48:50.078106 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:50.077922    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md2rf\" (UniqueName: \"kubernetes.io/projected/18bb436d-4071-4558-9428-9dfb7daebbe4-kube-api-access-md2rf\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z6dn4\" (UID: \"18bb436d-4071-4558-9428-9dfb7daebbe4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"
Oct 30 18:48:50.078267 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:50.078207    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18bb436d-4071-4558-9428-9dfb7daebbe4-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z6dn4\" (UID: \"18bb436d-4071-4558-9428-9dfb7daebbe4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"
Oct 30 18:48:50.078267 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:50.078230    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18bb436d-4071-4558-9428-9dfb7daebbe4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z6dn4\" (UID: \"18bb436d-4071-4558-9428-9dfb7daebbe4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"
Oct 30 18:48:50.078354 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:50.078305    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18bb436d-4071-4558-9428-9dfb7daebbe4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z6dn4\" (UID: \"18bb436d-4071-4558-9428-9dfb7daebbe4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"
Oct 30 18:48:50.080144 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:50.080125    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18bb436d-4071-4558-9428-9dfb7daebbe4-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z6dn4\" (UID: \"18bb436d-4071-4558-9428-9dfb7daebbe4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"
Oct 30 18:48:50.080397 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:50.080381    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18bb436d-4071-4558-9428-9dfb7daebbe4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z6dn4\" (UID: \"18bb436d-4071-4558-9428-9dfb7daebbe4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"
Oct 30 18:48:50.131708 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:50.131683    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md2rf\" (UniqueName: \"kubernetes.io/projected/18bb436d-4071-4558-9428-9dfb7daebbe4-kube-api-access-md2rf\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-z6dn4\" (UID: \"18bb436d-4071-4558-9428-9dfb7daebbe4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"
Oct 30 18:48:50.428440 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:50.428402    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"
Oct 30 18:48:50.787701 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:50.787604    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"]
Oct 30 18:48:50.789682 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:48:50.789658    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18bb436d_4071_4558_9428_9dfb7daebbe4.slice/crio-23be8d2ffd9279041d1b3397798f9adc9a214d82a286fe706f1a1eec488853fe WatchSource:0}: Error finding container 23be8d2ffd9279041d1b3397798f9adc9a214d82a286fe706f1a1eec488853fe: Status 404 returned error can't find the container with id 23be8d2ffd9279041d1b3397798f9adc9a214d82a286fe706f1a1eec488853fe
Oct 30 18:48:50.811226 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:50.811199    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4" event={"ID":"18bb436d-4071-4558-9428-9dfb7daebbe4","Type":"ContainerStarted","Data":"23be8d2ffd9279041d1b3397798f9adc9a214d82a286fe706f1a1eec488853fe"}
Oct 30 18:48:51.816827 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:51.816743    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4" event={"ID":"18bb436d-4071-4558-9428-9dfb7daebbe4","Type":"ContainerStarted","Data":"455a1f3313d934d3af837514e40125cfb71dad913ec6fb87d2a195df9419517c"}
Oct 30 18:48:51.929975 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:51.929927    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4" podStartSLOduration=2.359445145 podStartE2EDuration="2.929914275s" totalImagesPullingTime="570.46913ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:51.928461456 +0000 UTC m=+733.205511901" watchObservedRunningTime="2026-10-30 18:48:51.929914275 +0000 UTC m=+733.206964721"
Oct 30 18:48:52.428873 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:52.428842    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"
Oct 30 18:48:53.358090 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:53.358047    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:48:55.861786 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:55.861755    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"
Oct 30 18:48:55.873978 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:55.873955    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"
Oct 30 18:48:56.848397 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:56.848367    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-fd8zz"
Oct 30 18:48:58.107083 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.106963    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"]
Oct 30 18:48:58.111172 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.111144    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.153021 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.152971    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"]
Oct 30 18:48:58.156504 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.156484    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.161416 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.161386    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 18:48:58.242186 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.242150    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"]
Oct 30 18:48:58.246020 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.245996    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-g5j8l\""
Oct 30 18:48:58.255187 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.255157    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.255310 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.255209    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.255310 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.255287    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xfg7\" (UniqueName: \"kubernetes.io/projected/af172f82-2db0-4251-8669-ae5a1415c606-kube-api-access-6xfg7\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.255400 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.255352    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psp9w\" (UniqueName: \"kubernetes.io/projected/1e919a72-c6c1-4c1f-9e24-f7c5910021de-kube-api-access-psp9w\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.255400 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.255383    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af172f82-2db0-4251-8669-ae5a1415c606-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.255474 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.255417    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.255508 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.255475    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.255550 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.255521    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.255610 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.255593    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.255646 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.255631    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af172f82-2db0-4251-8669-ae5a1415c606-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.255691 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.255667    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.255741 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.255721    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.255783 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.255763    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.255831 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.255792    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.255885 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.255865    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.255921 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.255905    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.255985 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.255966    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.256041 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.256020    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af172f82-2db0-4251-8669-ae5a1415c606-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.317032 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.317002    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"]
Oct 30 18:48:58.357079 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.357037    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.357316 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.357101    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.357316 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.357125    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xfg7\" (UniqueName: \"kubernetes.io/projected/af172f82-2db0-4251-8669-ae5a1415c606-kube-api-access-6xfg7\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.357316 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.357169    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psp9w\" (UniqueName: \"kubernetes.io/projected/1e919a72-c6c1-4c1f-9e24-f7c5910021de-kube-api-access-psp9w\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.357316 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.357196    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af172f82-2db0-4251-8669-ae5a1415c606-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.357316 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.357228    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.357612 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.357459    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.357612 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.357512    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.357612 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.357543    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.357612 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.357567    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.357612 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.357571    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.357612 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.357580    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af172f82-2db0-4251-8669-ae5a1415c606-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.357612 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.357591    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.358040 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.357628    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.358040 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.357673    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.358040 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.357705    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.358040 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.357734    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.358040 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.357770    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.358040 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.357816    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.358040 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.357910    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.358040 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.357955    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.358040 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.357994    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af172f82-2db0-4251-8669-ae5a1415c606-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.358545 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.358087    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.358545 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.358193    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.358545 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.357946    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.358545 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.358282    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af172f82-2db0-4251-8669-ae5a1415c606-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.358740 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.358595    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.358740 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.358706    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.360352 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.360329    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.360679 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.360660    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.360770 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.360749    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.360816 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.360786    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af172f82-2db0-4251-8669-ae5a1415c606-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.411714 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.411686    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.411826 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.411714    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af172f82-2db0-4251-8669-ae5a1415c606-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.411890 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.411869    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xfg7\" (UniqueName: \"kubernetes.io/projected/af172f82-2db0-4251-8669-ae5a1415c606-kube-api-access-6xfg7\") pod \"maas-default-gateway-openshift-default-75d7c785f-9pgfc\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.412188 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.412172    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psp9w\" (UniqueName: \"kubernetes.io/projected/1e919a72-c6c1-4c1f-9e24-f7c5910021de-kube-api-access-psp9w\") pod \"maas-default-gateway-openshift-default-75d7c785f-f6zwn\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.450735 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.450714    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:48:58.471048 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.471025    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:48:58.864425 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.864400    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"]
Oct 30 18:48:58.866343 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:48:58.866313    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf172f82_2db0_4251_8669_ae5a1415c606.slice/crio-46529fefce8a54e0888c535d3db0f56e947848796709d71776ff7abafd246119 WatchSource:0}: Error finding container 46529fefce8a54e0888c535d3db0f56e947848796709d71776ff7abafd246119: Status 404 returned error can't find the container with id 46529fefce8a54e0888c535d3db0f56e947848796709d71776ff7abafd246119
Oct 30 18:48:58.867658 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:58.867634    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"]
Oct 30 18:48:58.901356 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:48:58.901326    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e919a72_c6c1_4c1f_9e24_f7c5910021de.slice/crio-ac2c27ae02cb0fa8acbdfa537b614eb5fd0936912c639d4aa3ac4dec1a35898f WatchSource:0}: Error finding container ac2c27ae02cb0fa8acbdfa537b614eb5fd0936912c639d4aa3ac4dec1a35898f: Status 404 returned error can't find the container with id ac2c27ae02cb0fa8acbdfa537b614eb5fd0936912c639d4aa3ac4dec1a35898f
Oct 30 18:48:59.853772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:59.853740    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn" event={"ID":"1e919a72-c6c1-4c1f-9e24-f7c5910021de","Type":"ContainerStarted","Data":"ac2c27ae02cb0fa8acbdfa537b614eb5fd0936912c639d4aa3ac4dec1a35898f"}
Oct 30 18:48:59.854933 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:48:59.854908    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc" event={"ID":"af172f82-2db0-4251-8669-ae5a1415c606","Type":"ContainerStarted","Data":"46529fefce8a54e0888c535d3db0f56e947848796709d71776ff7abafd246119"}
Oct 30 18:49:00.429552 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:00.429516    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"
Oct 30 18:49:00.442395 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:00.442366    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"
Oct 30 18:49:00.873523 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:00.873494    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-z6dn4"
Oct 30 18:49:03.174165 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:03.174134    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:49:03.174451 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:03.174199    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:49:03.174451 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:03.174228    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:49:03.877384 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:03.877351    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn" event={"ID":"1e919a72-c6c1-4c1f-9e24-f7c5910021de","Type":"ContainerStarted","Data":"361e8e91eb1c6cbbde4411a2bb7a06085f708e45d17b5facf349fa8c5b6e250e"}
Oct 30 18:49:04.080797 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:04.080743    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn" podStartSLOduration=1.810640778 podStartE2EDuration="6.080724385s" totalImagesPullingTime="4.270083607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:04.033199017 +0000 UTC m=+745.310249461" watchObservedRunningTime="2026-10-30 18:49:04.080724385 +0000 UTC m=+745.357774832"
Oct 30 18:49:04.275761 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:04.275728    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:49:04.276028 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:04.275793    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:49:04.276028 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:04.275824    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:49:04.451200 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:04.451156    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:49:04.452717 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:04.452609    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-f6zwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 30 18:49:04.452717 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:04.452665    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn" podUID="1e919a72-c6c1-4c1f-9e24-f7c5910021de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 30 18:49:04.889263 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:04.889198    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc" event={"ID":"af172f82-2db0-4251-8669-ae5a1415c606","Type":"ContainerStarted","Data":"fef22f044647d3caab513979d7fea0d7a59c9bf3255e4f5af66fb59dd702066c"}
Oct 30 18:49:05.024912 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:05.024540    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc" podStartSLOduration=1.617964775 podStartE2EDuration="7.024520027s" totalImagesPullingTime="5.406555252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:05.02444254 +0000 UTC m=+746.301492987" watchObservedRunningTime="2026-10-30 18:49:05.024520027 +0000 UTC m=+746.301570473"
Oct 30 18:49:05.453564 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:05.453520    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-f6zwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 30 18:49:05.454136 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:05.453603    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn" podUID="1e919a72-c6c1-4c1f-9e24-f7c5910021de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 30 18:49:05.471728 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:05.471699    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:49:05.473280 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:05.473256    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9pgfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Oct 30 18:49:05.473394 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:05.473313    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc" podUID="af172f82-2db0-4251-8669-ae5a1415c606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Oct 30 18:49:06.451484 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:06.451451    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-f6zwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 30 18:49:06.451718 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:06.451698    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn" podUID="1e919a72-c6c1-4c1f-9e24-f7c5910021de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 30 18:49:06.471830 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:06.471797    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9pgfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Oct 30 18:49:06.472224 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:06.471855    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc" podUID="af172f82-2db0-4251-8669-ae5a1415c606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Oct 30 18:49:07.451703 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:07.451651    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-f6zwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 30 18:49:07.451922 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:07.451747    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn" podUID="1e919a72-c6c1-4c1f-9e24-f7c5910021de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 30 18:49:07.471649 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:07.471614    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9pgfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Oct 30 18:49:07.471784 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:07.471664    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc" podUID="af172f82-2db0-4251-8669-ae5a1415c606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Oct 30 18:49:08.451612 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:08.451571    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-f6zwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 30 18:49:08.452265 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:08.452235    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn" podUID="1e919a72-c6c1-4c1f-9e24-f7c5910021de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 30 18:49:08.472270 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:08.472244    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9pgfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Oct 30 18:49:08.472403 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:08.472286    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc" podUID="af172f82-2db0-4251-8669-ae5a1415c606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Oct 30 18:49:09.451242 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:09.451195    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-f6zwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 30 18:49:09.451459 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:09.451264    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn" podUID="1e919a72-c6c1-4c1f-9e24-f7c5910021de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 30 18:49:09.471517 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:09.471485    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9pgfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Oct 30 18:49:09.471980 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:09.471538    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc" podUID="af172f82-2db0-4251-8669-ae5a1415c606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Oct 30 18:49:10.451265 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:10.451230    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-f6zwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 30 18:49:10.451499 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:10.451285    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn" podUID="1e919a72-c6c1-4c1f-9e24-f7c5910021de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 30 18:49:10.471530 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:10.471499    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9pgfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Oct 30 18:49:10.471955 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:10.471533    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc" podUID="af172f82-2db0-4251-8669-ae5a1415c606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Oct 30 18:49:11.471831 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:11.471770    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9pgfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body=
Oct 30 18:49:11.472339 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:11.471863    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc" podUID="af172f82-2db0-4251-8669-ae5a1415c606" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused"
Oct 30 18:49:11.660187 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:11.660123    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:49:11.660651 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:11.660627    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:49:11.661455 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:11.661435    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:49:12.672811 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:12.672777    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:49:12.673293 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:12.673266    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:49:12.673930 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:12.673910    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:49:15.710414 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:49:15.710380    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:49:16.999325 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:16.999283    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-575997bf46-rbd6m"]
Oct 30 18:49:17.005704 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:17.005679    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-575997bf46-rbd6m"
Oct 30 18:49:17.057187 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:17.057158    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-575997bf46-rbd6m"]
Oct 30 18:49:17.065449 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:17.065423    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ea148539-543a-45f7-9a45-9351959b8b6d-tls-cert\") pod \"authorino-575997bf46-rbd6m\" (UID: \"ea148539-543a-45f7-9a45-9351959b8b6d\") " pod="kuadrant-system/authorino-575997bf46-rbd6m"
Oct 30 18:49:17.065579 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:17.065463    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mdpb\" (UniqueName: \"kubernetes.io/projected/ea148539-543a-45f7-9a45-9351959b8b6d-kube-api-access-8mdpb\") pod \"authorino-575997bf46-rbd6m\" (UID: \"ea148539-543a-45f7-9a45-9351959b8b6d\") " pod="kuadrant-system/authorino-575997bf46-rbd6m"
Oct 30 18:49:17.166166 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:17.166138    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ea148539-543a-45f7-9a45-9351959b8b6d-tls-cert\") pod \"authorino-575997bf46-rbd6m\" (UID: \"ea148539-543a-45f7-9a45-9351959b8b6d\") " pod="kuadrant-system/authorino-575997bf46-rbd6m"
Oct 30 18:49:17.166325 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:17.166185    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mdpb\" (UniqueName: \"kubernetes.io/projected/ea148539-543a-45f7-9a45-9351959b8b6d-kube-api-access-8mdpb\") pod \"authorino-575997bf46-rbd6m\" (UID: \"ea148539-543a-45f7-9a45-9351959b8b6d\") " pod="kuadrant-system/authorino-575997bf46-rbd6m"
Oct 30 18:49:17.168454 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:17.168427    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ea148539-543a-45f7-9a45-9351959b8b6d-tls-cert\") pod \"authorino-575997bf46-rbd6m\" (UID: \"ea148539-543a-45f7-9a45-9351959b8b6d\") " pod="kuadrant-system/authorino-575997bf46-rbd6m"
Oct 30 18:49:17.221460 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:17.221432    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mdpb\" (UniqueName: \"kubernetes.io/projected/ea148539-543a-45f7-9a45-9351959b8b6d-kube-api-access-8mdpb\") pod \"authorino-575997bf46-rbd6m\" (UID: \"ea148539-543a-45f7-9a45-9351959b8b6d\") " pod="kuadrant-system/authorino-575997bf46-rbd6m"
Oct 30 18:49:17.250298 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:17.250199    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:49:17.318993 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:17.318962    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-575997bf46-rbd6m"
Oct 30 18:49:17.474914 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:17.474883    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:49:17.682279 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:17.682255    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-575997bf46-rbd6m"]
Oct 30 18:49:17.684868 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:49:17.684835    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea148539_543a_45f7_9a45_9351959b8b6d.slice/crio-2c1ba2c684d1b639e8255951c2544608948b2f9a547a3a3537d7f15a99806e71 WatchSource:0}: Error finding container 2c1ba2c684d1b639e8255951c2544608948b2f9a547a3a3537d7f15a99806e71: Status 404 returned error can't find the container with id 2c1ba2c684d1b639e8255951c2544608948b2f9a547a3a3537d7f15a99806e71
Oct 30 18:49:17.993028 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:17.992949    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-575997bf46-rbd6m" event={"ID":"ea148539-543a-45f7-9a45-9351959b8b6d","Type":"ContainerStarted","Data":"2c1ba2c684d1b639e8255951c2544608948b2f9a547a3a3537d7f15a99806e71"}
Oct 30 18:49:20.003407 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:20.003321    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-575997bf46-rbd6m" event={"ID":"ea148539-543a-45f7-9a45-9351959b8b6d","Type":"ContainerStarted","Data":"af814c95480dce485589fccc29e32378cbd4f7bbb3fe094f885b3c759e209114"}
Oct 30 18:49:20.109982 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:20.109933    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-575997bf46-rbd6m" podStartSLOduration=2.048814205 podStartE2EDuration="4.109922091s" totalImagesPullingTime="2.061107886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:20.109755144 +0000 UTC m=+761.386805588" watchObservedRunningTime="2026-10-30 18:49:20.109922091 +0000 UTC m=+761.386972537"
Oct 30 18:49:20.158284 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:20.158250    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f955ff69c-rmvbv"]
Oct 30 18:49:20.158468 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:20.158440    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5f955ff69c-rmvbv" podUID="d09b131f-77e0-4e2b-a529-d7e0f5ab33a8" containerName="authorino" containerID="cri-o://6c4190b4b4cc568ea4c73a2837aa7a443fde4a93e6acbb974fa55aaa5ce1138f" gracePeriod=30
Oct 30 18:49:20.406443 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:20.406415    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f955ff69c-rmvbv"
Oct 30 18:49:20.496180 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:20.496143    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d09b131f-77e0-4e2b-a529-d7e0f5ab33a8-kube-api-access-85kcl\" (UniqueName: \"kubernetes.io/projected/d09b131f-77e0-4e2b-a529-d7e0f5ab33a8-kube-api-access-85kcl\") pod \"d09b131f-77e0-4e2b-a529-d7e0f5ab33a8\" (UID: \"d09b131f-77e0-4e2b-a529-d7e0f5ab33a8\") "
Oct 30 18:49:20.496349 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:20.496196    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d09b131f-77e0-4e2b-a529-d7e0f5ab33a8-tls-cert\" (UniqueName: \"kubernetes.io/secret/d09b131f-77e0-4e2b-a529-d7e0f5ab33a8-tls-cert\") pod \"d09b131f-77e0-4e2b-a529-d7e0f5ab33a8\" (UID: \"d09b131f-77e0-4e2b-a529-d7e0f5ab33a8\") "
Oct 30 18:49:20.498218 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:20.498188    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d09b131f-77e0-4e2b-a529-d7e0f5ab33a8-kube-api-access-85kcl" pod "d09b131f-77e0-4e2b-a529-d7e0f5ab33a8" (UID: "d09b131f-77e0-4e2b-a529-d7e0f5ab33a8"). InnerVolumeSpecName "kube-api-access-85kcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:49:20.507315 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:20.507282    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d09b131f-77e0-4e2b-a529-d7e0f5ab33a8-tls-cert" pod "d09b131f-77e0-4e2b-a529-d7e0f5ab33a8" (UID: "d09b131f-77e0-4e2b-a529-d7e0f5ab33a8"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:49:20.599447 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:20.599410    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85kcl\" (UniqueName: \"kubernetes.io/projected/d09b131f-77e0-4e2b-a529-d7e0f5ab33a8-kube-api-access-85kcl\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:20.599447 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:20.599444    4924 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d09b131f-77e0-4e2b-a529-d7e0f5ab33a8-tls-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:49:21.009805 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:21.009731    4924 generic.go:356] "Generic (PLEG): container finished" podID="d09b131f-77e0-4e2b-a529-d7e0f5ab33a8" containerID="6c4190b4b4cc568ea4c73a2837aa7a443fde4a93e6acbb974fa55aaa5ce1138f" exitCode=0
Oct 30 18:49:21.009805 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:21.009774    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f955ff69c-rmvbv"
Oct 30 18:49:21.010282 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:21.009812    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f955ff69c-rmvbv" event={"ID":"d09b131f-77e0-4e2b-a529-d7e0f5ab33a8","Type":"ContainerDied","Data":"6c4190b4b4cc568ea4c73a2837aa7a443fde4a93e6acbb974fa55aaa5ce1138f"}
Oct 30 18:49:21.010282 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:21.009856    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f955ff69c-rmvbv" event={"ID":"d09b131f-77e0-4e2b-a529-d7e0f5ab33a8","Type":"ContainerDied","Data":"e7cc9e0fbd862525081c6a9d82ee92011274fb6053720d766d146fbcbcc1afd5"}
Oct 30 18:49:21.010282 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:21.009877    4924 scope.go:122] "RemoveContainer" containerID="6c4190b4b4cc568ea4c73a2837aa7a443fde4a93e6acbb974fa55aaa5ce1138f"
Oct 30 18:49:21.018392 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:21.018367    4924 scope.go:122] "RemoveContainer" containerID="6c4190b4b4cc568ea4c73a2837aa7a443fde4a93e6acbb974fa55aaa5ce1138f"
Oct 30 18:49:21.018659 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:49:21.018639    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c4190b4b4cc568ea4c73a2837aa7a443fde4a93e6acbb974fa55aaa5ce1138f\": container with ID starting with 6c4190b4b4cc568ea4c73a2837aa7a443fde4a93e6acbb974fa55aaa5ce1138f not found: ID does not exist" containerID="6c4190b4b4cc568ea4c73a2837aa7a443fde4a93e6acbb974fa55aaa5ce1138f"
Oct 30 18:49:21.018710 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:21.018668    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c4190b4b4cc568ea4c73a2837aa7a443fde4a93e6acbb974fa55aaa5ce1138f"} err="failed to get container status \"6c4190b4b4cc568ea4c73a2837aa7a443fde4a93e6acbb974fa55aaa5ce1138f\": rpc error: code = NotFound desc = could not find container \"6c4190b4b4cc568ea4c73a2837aa7a443fde4a93e6acbb974fa55aaa5ce1138f\": container with ID starting with 6c4190b4b4cc568ea4c73a2837aa7a443fde4a93e6acbb974fa55aaa5ce1138f not found: ID does not exist"
Oct 30 18:49:21.185951 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:21.185916    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f955ff69c-rmvbv"]
Oct 30 18:49:21.187038 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:21.187006    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f955ff69c-rmvbv"]
Oct 30 18:49:22.832541 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:22.832499    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d09b131f-77e0-4e2b-a529-d7e0f5ab33a8" path="/var/lib/kubelet/pods/d09b131f-77e0-4e2b-a529-d7e0f5ab33a8/volumes"
Oct 30 18:49:27.462056 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:27.462025    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:49:27.684040 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:27.684005    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:49:27.846338 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:27.846309    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-lcgr2"]
Oct 30 18:49:27.846847 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:27.846833    4924 cpu_manager.go:414] "containerMap: removing container" podUID="d09b131f-77e0-4e2b-a529-d7e0f5ab33a8" containerName="authorino"
Oct 30 18:49:27.846903 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:27.846849    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d09b131f-77e0-4e2b-a529-d7e0f5ab33a8" containerName="authorino"
Oct 30 18:49:27.846942 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:27.846935    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d09b131f-77e0-4e2b-a529-d7e0f5ab33a8" containerName="authorino"
Oct 30 18:49:27.850363 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:27.850340    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lcgr2"
Oct 30 18:49:27.903758 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:27.903728    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-lcgr2"]
Oct 30 18:49:27.947727 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:27.947698    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-5jsvm\""
Oct 30 18:49:27.965095 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:27.965047    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ndps\" (UniqueName: \"kubernetes.io/projected/589c0305-3c1e-48d9-ad0a-7897628d9b57-kube-api-access-6ndps\") pod \"payload-pre-processing-7b98bb678c-lcgr2\" (UID: \"589c0305-3c1e-48d9-ad0a-7897628d9b57\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-lcgr2"
Oct 30 18:49:27.965095 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:27.965049    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 18:49:27.965313 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:27.965227    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589c0305-3c1e-48d9-ad0a-7897628d9b57-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-lcgr2\" (UID: \"589c0305-3c1e-48d9-ad0a-7897628d9b57\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-lcgr2"
Oct 30 18:49:27.995865 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:27.995833    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-52rkw"]
Oct 30 18:49:27.999310 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:27.999293    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-52rkw"
Oct 30 18:49:28.052965 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:28.051764    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-52rkw"]
Oct 30 18:49:28.065905 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:28.065875    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589c0305-3c1e-48d9-ad0a-7897628d9b57-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-lcgr2\" (UID: \"589c0305-3c1e-48d9-ad0a-7897628d9b57\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-lcgr2"
Oct 30 18:49:28.066085 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:28.065943    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ndps\" (UniqueName: \"kubernetes.io/projected/589c0305-3c1e-48d9-ad0a-7897628d9b57-kube-api-access-6ndps\") pod \"payload-pre-processing-7b98bb678c-lcgr2\" (UID: \"589c0305-3c1e-48d9-ad0a-7897628d9b57\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-lcgr2"
Oct 30 18:49:28.066085 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:28.066007    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221a2c35-e283-4aef-8289-ebecb034e652-plugins-config-volume\") pod \"payload-processing-5f78687d5b-52rkw\" (UID: \"221a2c35-e283-4aef-8289-ebecb034e652\") " pod="openshift-ingress/payload-processing-5f78687d5b-52rkw"
Oct 30 18:49:28.066085 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:28.066040    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbx8f\" (UniqueName: \"kubernetes.io/projected/221a2c35-e283-4aef-8289-ebecb034e652-kube-api-access-gbx8f\") pod \"payload-processing-5f78687d5b-52rkw\" (UID: \"221a2c35-e283-4aef-8289-ebecb034e652\") " pod="openshift-ingress/payload-processing-5f78687d5b-52rkw"
Oct 30 18:49:28.066674 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:28.066650    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589c0305-3c1e-48d9-ad0a-7897628d9b57-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-lcgr2\" (UID: \"589c0305-3c1e-48d9-ad0a-7897628d9b57\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-lcgr2"
Oct 30 18:49:28.120398 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:28.120326    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ndps\" (UniqueName: \"kubernetes.io/projected/589c0305-3c1e-48d9-ad0a-7897628d9b57-kube-api-access-6ndps\") pod \"payload-pre-processing-7b98bb678c-lcgr2\" (UID: \"589c0305-3c1e-48d9-ad0a-7897628d9b57\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-lcgr2"
Oct 30 18:49:28.162703 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:28.162679    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lcgr2"
Oct 30 18:49:28.167173 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:28.167150    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221a2c35-e283-4aef-8289-ebecb034e652-plugins-config-volume\") pod \"payload-processing-5f78687d5b-52rkw\" (UID: \"221a2c35-e283-4aef-8289-ebecb034e652\") " pod="openshift-ingress/payload-processing-5f78687d5b-52rkw"
Oct 30 18:49:28.167276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:28.167190    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbx8f\" (UniqueName: \"kubernetes.io/projected/221a2c35-e283-4aef-8289-ebecb034e652-kube-api-access-gbx8f\") pod \"payload-processing-5f78687d5b-52rkw\" (UID: \"221a2c35-e283-4aef-8289-ebecb034e652\") " pod="openshift-ingress/payload-processing-5f78687d5b-52rkw"
Oct 30 18:49:28.168075 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:28.168035    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221a2c35-e283-4aef-8289-ebecb034e652-plugins-config-volume\") pod \"payload-processing-5f78687d5b-52rkw\" (UID: \"221a2c35-e283-4aef-8289-ebecb034e652\") " pod="openshift-ingress/payload-processing-5f78687d5b-52rkw"
Oct 30 18:49:28.222167 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:28.222135    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbx8f\" (UniqueName: \"kubernetes.io/projected/221a2c35-e283-4aef-8289-ebecb034e652-kube-api-access-gbx8f\") pod \"payload-processing-5f78687d5b-52rkw\" (UID: \"221a2c35-e283-4aef-8289-ebecb034e652\") " pod="openshift-ingress/payload-processing-5f78687d5b-52rkw"
Oct 30 18:49:28.315796 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:28.315756    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-52rkw"
Oct 30 18:49:28.533846 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:28.533811    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-lcgr2"]
Oct 30 18:49:28.536570 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:49:28.536534    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod589c0305_3c1e_48d9_ad0a_7897628d9b57.slice/crio-f695f6323dae132df0d58075c1e788dd8f9d71f167f134c2306f1b64ef7bb646 WatchSource:0}: Error finding container f695f6323dae132df0d58075c1e788dd8f9d71f167f134c2306f1b64ef7bb646: Status 404 returned error can't find the container with id f695f6323dae132df0d58075c1e788dd8f9d71f167f134c2306f1b64ef7bb646
Oct 30 18:49:28.683149 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:28.683051    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-52rkw"]
Oct 30 18:49:28.686317 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:49:28.686286    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod221a2c35_e283_4aef_8289_ebecb034e652.slice/crio-c4d20d526bd67b00494531e8b4020166b263f163457b1f68270a7701a1f0fb2d WatchSource:0}: Error finding container c4d20d526bd67b00494531e8b4020166b263f163457b1f68270a7701a1f0fb2d: Status 404 returned error can't find the container with id c4d20d526bd67b00494531e8b4020166b263f163457b1f68270a7701a1f0fb2d
Oct 30 18:49:29.050141 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:29.050025    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lcgr2" event={"ID":"589c0305-3c1e-48d9-ad0a-7897628d9b57","Type":"ContainerStarted","Data":"f695f6323dae132df0d58075c1e788dd8f9d71f167f134c2306f1b64ef7bb646"}
Oct 30 18:49:29.051876 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:29.051834    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-52rkw" event={"ID":"221a2c35-e283-4aef-8289-ebecb034e652","Type":"ContainerStarted","Data":"c4d20d526bd67b00494531e8b4020166b263f163457b1f68270a7701a1f0fb2d"}
Oct 30 18:49:32.570239 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:32.570148    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:49:33.076951 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:33.076913    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lcgr2" event={"ID":"589c0305-3c1e-48d9-ad0a-7897628d9b57","Type":"ContainerStarted","Data":"87af4acc207d749bd5a82fc91ab5742a347c46838d88cf2ddbe5370309b42bb9"}
Oct 30 18:49:33.077154 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:33.077083    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lcgr2"
Oct 30 18:49:33.186556 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:33.186495    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lcgr2" podStartSLOduration=2.431479541 podStartE2EDuration="6.18647632s" totalImagesPullingTime="3.754996779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:33.183769989 +0000 UTC m=+774.460820433" watchObservedRunningTime="2026-10-30 18:49:33.18647632 +0000 UTC m=+774.463526766"
Oct 30 18:49:37.096155 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:37.096110    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-52rkw" event={"ID":"221a2c35-e283-4aef-8289-ebecb034e652","Type":"ContainerStarted","Data":"124d0ca41327b6b3830020628063d417afd0b76c2dd1a6fbc2699e6814815b44"}
Oct 30 18:49:37.096621 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:37.096177    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-52rkw"
Oct 30 18:49:37.205362 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:37.205310    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-52rkw" podStartSLOduration=2.516011411 podStartE2EDuration="10.205294492s" totalImagesPullingTime="7.689283081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:37.203612362 +0000 UTC m=+778.480662804" watchObservedRunningTime="2026-10-30 18:49:37.205294492 +0000 UTC m=+778.482344936"
Oct 30 18:49:44.084590 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:44.084565    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lcgr2"
Oct 30 18:49:45.710661 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:49:45.710619    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:49:48.103298 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:48.103264    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-52rkw"
Oct 30 18:49:51.538757 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:51.538721    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:49:52.079937 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:52.079904    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:49:53.713138 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:53.713109    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:49:53.972555 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:53.972480    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:49:55.342043 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:55.342007    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:49:55.533934 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:49:55.533894    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:15.710474 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:50:15.710430    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:50:21.250740 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:21.250710    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:21.446010 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:21.445978    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:21.970350 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:21.970314    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"]
Oct 30 18:50:21.975326 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:21.975304    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.024738 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.024707    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-649d16-openshift-default-dockercfg-xgvdr\""
Oct 30 18:50:22.063661 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.063630    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-workload-certs\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.063661 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.063663    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-istio-envoy\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.063859 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.063682    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-istio-data\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.063859 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.063733    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53aa846-ec01-4b44-a518-49a558965906-istiod-ca-cert\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.063859 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.063769    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53aa846-ec01-4b44-a518-49a558965906-istio-podinfo\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.063859 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.063820    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53aa846-ec01-4b44-a518-49a558965906-istio-token\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.063980 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.063885    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-workload-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.063980 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.063902    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-credential-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.063980 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.063928    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwzxl\" (UniqueName: \"kubernetes.io/projected/f53aa846-ec01-4b44-a518-49a558965906-kube-api-access-cwzxl\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.072465 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.072442    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"]
Oct 30 18:50:22.116899 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.116868    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:22.165320 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.165291    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-workload-certs\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.165481 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.165328    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-istio-envoy\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.165481 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.165355    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-istio-data\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.165481 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.165373    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53aa846-ec01-4b44-a518-49a558965906-istiod-ca-cert\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.165481 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.165396    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53aa846-ec01-4b44-a518-49a558965906-istio-podinfo\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.165481 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.165426    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53aa846-ec01-4b44-a518-49a558965906-istio-token\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.165728 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.165519    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-workload-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.165728 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.165549    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-credential-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.165728 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.165585    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwzxl\" (UniqueName: \"kubernetes.io/projected/f53aa846-ec01-4b44-a518-49a558965906-kube-api-access-cwzxl\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.165871 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.165793    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-workload-certs\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.165871 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.165809    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-istio-data\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.165972 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.165908    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-workload-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.166145 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.166122    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-credential-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.166260 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.166240    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53aa846-ec01-4b44-a518-49a558965906-istiod-ca-cert\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.167820 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.167798    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-istio-envoy\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.167994 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.167974    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53aa846-ec01-4b44-a518-49a558965906-istio-podinfo\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.224015 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.223952    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53aa846-ec01-4b44-a518-49a558965906-istio-token\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.224147 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.224027    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwzxl\" (UniqueName: \"kubernetes.io/projected/f53aa846-ec01-4b44-a518-49a558965906-kube-api-access-cwzxl\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.286529 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.286492    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:22.339753 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.339724    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:22.550230 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.550157    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:22.627093 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.627050    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"]
Oct 30 18:50:22.629973 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.629953    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.684272 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.684239    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-3b6ad6-openshift-default-dockercfg-wmgsn\""
Oct 30 18:50:22.691848 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.691819    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"]
Oct 30 18:50:22.703884 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.703862    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"]
Oct 30 18:50:22.705901 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:50:22.705878    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf53aa846_ec01_4b44_a518_49a558965906.slice/crio-ceebc51234837bfe3a9bc708468043ad6c31b49abce9cc097ae962a12448e578 WatchSource:0}: Error finding container ceebc51234837bfe3a9bc708468043ad6c31b49abce9cc097ae962a12448e578: Status 404 returned error can't find the container with id ceebc51234837bfe3a9bc708468043ad6c31b49abce9cc097ae962a12448e578
Oct 30 18:50:22.708154 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.708124    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:50:22.708222 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.708205    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:50:22.708269 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.708234    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:50:22.755947 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.755926    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:22.770723 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.770700    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-credential-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.770824 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.770751    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e73c307-868e-40e8-ad04-099a7446ccef-istiod-ca-cert\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.770876 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.770823    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e73c307-868e-40e8-ad04-099a7446ccef-istio-podinfo\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.770931 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.770914    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj76d\" (UniqueName: \"kubernetes.io/projected/1e73c307-868e-40e8-ad04-099a7446ccef-kube-api-access-fj76d\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.770971 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.770950    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-workload-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.771007 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.770978    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-workload-certs\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.771046 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.771008    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-istio-envoy\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.771099 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.771054    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e73c307-868e-40e8-ad04-099a7446ccef-istio-token\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.771135 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.771113    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-istio-data\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.872018 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.871991    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-istio-data\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.872222 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.872034    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-credential-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.872222 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.872059    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e73c307-868e-40e8-ad04-099a7446ccef-istiod-ca-cert\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.872222 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.872112    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e73c307-868e-40e8-ad04-099a7446ccef-istio-podinfo\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.872222 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.872168    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj76d\" (UniqueName: \"kubernetes.io/projected/1e73c307-868e-40e8-ad04-099a7446ccef-kube-api-access-fj76d\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.872222 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.872193    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-workload-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.872222 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.872215    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-workload-certs\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.872551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.872255    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-istio-envoy\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.872551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.872297    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e73c307-868e-40e8-ad04-099a7446ccef-istio-token\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.872551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.872440    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-istio-data\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.872711 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.872676    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-workload-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.872775 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.872727    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-workload-certs\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.872775 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.872728    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-credential-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.873158 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.873138    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e73c307-868e-40e8-ad04-099a7446ccef-istiod-ca-cert\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.874501 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.874480    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-istio-envoy\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.874729 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.874713    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e73c307-868e-40e8-ad04-099a7446ccef-istio-podinfo\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.928032 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.927999    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e73c307-868e-40e8-ad04-099a7446ccef-istio-token\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.928200 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.928184    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj76d\" (UniqueName: \"kubernetes.io/projected/1e73c307-868e-40e8-ad04-099a7446ccef-kube-api-access-fj76d\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.942500 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.942479    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:22.971271 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:22.971231    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:23.172046 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.171942    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:23.294464 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.294430    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"]
Oct 30 18:50:23.297284 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.297268    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.310747 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.310714    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g" event={"ID":"f53aa846-ec01-4b44-a518-49a558965906","Type":"ContainerStarted","Data":"b7ed64f2a0ffae646e5fd3f7e0d50c1ce291c58ad8d2678edbae7d7165eaf89a"}
Oct 30 18:50:23.310894 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.310792    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g" event={"ID":"f53aa846-ec01-4b44-a518-49a558965906","Type":"ContainerStarted","Data":"ceebc51234837bfe3a9bc708468043ad6c31b49abce9cc097ae962a12448e578"}
Oct 30 18:50:23.347421 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.347388    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f5705f-openshift-default-dockercfg-hwfjh\""
Oct 30 18:50:23.359505 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.359476    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"]
Oct 30 18:50:23.371277 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.371196    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"]
Oct 30 18:50:23.373777 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:50:23.373748    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e73c307_868e_40e8_ad04_099a7446ccef.slice/crio-6498b507b4a3daf039ad4268e7735c6a1f04ebb95a3b17f7e74961e3338c2d32 WatchSource:0}: Error finding container 6498b507b4a3daf039ad4268e7735c6a1f04ebb95a3b17f7e74961e3338c2d32: Status 404 returned error can't find the container with id 6498b507b4a3daf039ad4268e7735c6a1f04ebb95a3b17f7e74961e3338c2d32
Oct 30 18:50:23.376175 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.376152    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:23.376516 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.376485    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:50:23.376568 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.376550    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:50:23.376599 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.376581    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:50:23.479522 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.479485    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-envoy\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.479696 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.479545    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh62c\" (UniqueName: \"kubernetes.io/projected/ccddacc3-f62d-477a-9c8b-9ead25b237c4-kube-api-access-bh62c\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.479696 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.479682    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-data\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.479861 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.479839    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-workload-certs\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.479977 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.479957    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-workload-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.480037 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.479991    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istiod-ca-cert\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.480112 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.480083    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-token\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.480179 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.480161    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-credential-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.480248 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.480227    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-podinfo\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.528100 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.528025    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g" podStartSLOduration=2.5280125509999998 podStartE2EDuration="2.528012551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:23.525350902 +0000 UTC m=+824.802401338" watchObservedRunningTime="2026-10-30 18:50:23.528012551 +0000 UTC m=+824.805062996"
Oct 30 18:50:23.580888 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.580852    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-workload-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.580888 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.580888    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istiod-ca-cert\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.581146 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.580908    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-token\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.581146 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.581124    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-credential-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.581248 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.581182    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-podinfo\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.581248 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.581225    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-envoy\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.581356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.581268    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh62c\" (UniqueName: \"kubernetes.io/projected/ccddacc3-f62d-477a-9c8b-9ead25b237c4-kube-api-access-bh62c\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.581356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.581299    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-data\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.581457 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.581383    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-workload-certs\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.581457 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.581429    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-workload-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.581712 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.581685    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istiod-ca-cert\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.581803 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.581776    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-data\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.581871 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.581854    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-credential-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.582118 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.582093    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-workload-certs\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.584123 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.584097    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-podinfo\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.584298 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.584277    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-envoy\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.589527 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.589501    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:23.640980 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.640953    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh62c\" (UniqueName: \"kubernetes.io/projected/ccddacc3-f62d-477a-9c8b-9ead25b237c4-kube-api-access-bh62c\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.644785 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.644763    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-token\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.916023 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.915996    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:23.953918 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:23.953822    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:24.000093 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.000045    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"]
Oct 30 18:50:24.003138 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.003114    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.055723 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.055697    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-a1617e-openshift-default-dockercfg-fbsx5\""
Oct 30 18:50:24.063693 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.063661    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"]
Oct 30 18:50:24.155210 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.155163    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:24.187492 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.187426    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-workload-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.187492 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.187483    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz4cd\" (UniqueName: \"kubernetes.io/projected/d6ec64bd-1810-4f78-b451-9faae648be6e-kube-api-access-wz4cd\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.187669 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.187578    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-credential-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.187669 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.187634    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6ec64bd-1810-4f78-b451-9faae648be6e-istiod-ca-cert\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.187669 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.187652    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-data\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.187772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.187676    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-envoy\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.187772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.187693    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-podinfo\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.187839 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.187793    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-token\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.187839 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.187834    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-workload-certs\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.281994 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.281967    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"]
Oct 30 18:50:24.284877 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:50:24.284849    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccddacc3_f62d_477a_9c8b_9ead25b237c4.slice/crio-b1aac27ae395ea94c97ce83835f80d4ea16eda598ac9e80adb4a71dc488a39ec WatchSource:0}: Error finding container b1aac27ae395ea94c97ce83835f80d4ea16eda598ac9e80adb4a71dc488a39ec: Status 404 returned error can't find the container with id b1aac27ae395ea94c97ce83835f80d4ea16eda598ac9e80adb4a71dc488a39ec
Oct 30 18:50:24.286883 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.286861    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:24.287260 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.287230    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:50:24.287360 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.287291    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:50:24.287360 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.287319    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:50:24.288294 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.288276    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz4cd\" (UniqueName: \"kubernetes.io/projected/d6ec64bd-1810-4f78-b451-9faae648be6e-kube-api-access-wz4cd\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.288401 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.288306    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-credential-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.288401 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.288336    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6ec64bd-1810-4f78-b451-9faae648be6e-istiod-ca-cert\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.288401 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.288359    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-data\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.288401 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.288394    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-envoy\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.288613 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.288418    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-podinfo\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.288613 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.288448    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-token\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.288613 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.288491    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-workload-certs\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.288613 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.288550    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-workload-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.288817 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.288703    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-credential-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.289207 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.289181    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-data\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.289431 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.289410    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-workload-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.289534 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.289512    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-workload-certs\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.289678 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.289576    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6ec64bd-1810-4f78-b451-9faae648be6e-istiod-ca-cert\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.291472 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.291452    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-podinfo\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.291590 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.291497    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-envoy\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.293715 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.293530    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:24.318154 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.318124    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv" event={"ID":"ccddacc3-f62d-477a-9c8b-9ead25b237c4","Type":"ContainerStarted","Data":"b1aac27ae395ea94c97ce83835f80d4ea16eda598ac9e80adb4a71dc488a39ec"}
Oct 30 18:50:24.319919 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.319890    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f" event={"ID":"1e73c307-868e-40e8-ad04-099a7446ccef","Type":"ContainerStarted","Data":"a3e84af7e790444c71aebec9462ee8037428558eaad1a91b9c35353b317a238e"}
Oct 30 18:50:24.320034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.319938    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f" event={"ID":"1e73c307-868e-40e8-ad04-099a7446ccef","Type":"ContainerStarted","Data":"6498b507b4a3daf039ad4268e7735c6a1f04ebb95a3b17f7e74961e3338c2d32"}
Oct 30 18:50:24.320131 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.320038    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:24.321143 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.321124    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:50:24.342797 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.342774    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-token\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.343803 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.343783    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz4cd\" (UniqueName: \"kubernetes.io/projected/d6ec64bd-1810-4f78-b451-9faae648be6e-kube-api-access-wz4cd\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.373122 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.373091    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:24.615514 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.615482    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:24.637867 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.637821    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f" podStartSLOduration=2.6378066970000003 podStartE2EDuration="2.637806697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:24.636121376 +0000 UTC m=+825.913171823" watchObservedRunningTime="2026-10-30 18:50:24.637806697 +0000 UTC m=+825.914857142"
Oct 30 18:50:24.942908 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.942837    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:24.947846 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.947821    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:24.983342 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.983322    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"]
Oct 30 18:50:24.985043 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:50:24.985016    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ec64bd_1810_4f78_b451_9faae648be6e.slice/crio-ccb204b15c58d4a332d812d15466088da075a18345ffa09daa590018cbb0f791 WatchSource:0}: Error finding container ccb204b15c58d4a332d812d15466088da075a18345ffa09daa590018cbb0f791: Status 404 returned error can't find the container with id ccb204b15c58d4a332d812d15466088da075a18345ffa09daa590018cbb0f791
Oct 30 18:50:24.987485 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.987456    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:50:24.987580 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.987528    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:50:24.987580 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.987561    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:50:24.988122 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:24.988028    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:25.326868 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:25.326770    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv" event={"ID":"ccddacc3-f62d-477a-9c8b-9ead25b237c4","Type":"ContainerStarted","Data":"24edc2c8cc0a4bd1289895d4ca56ccc2dfe7b7f1310385b5f6bea38b7d9b1871"}
Oct 30 18:50:25.328830 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:25.328799    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w" event={"ID":"d6ec64bd-1810-4f78-b451-9faae648be6e","Type":"ContainerStarted","Data":"8368c268d84e8ae8b1654db113e7462130487deb364760b85708c6aa49205751"}
Oct 30 18:50:25.328960 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:25.328855    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w" event={"ID":"d6ec64bd-1810-4f78-b451-9faae648be6e","Type":"ContainerStarted","Data":"ccb204b15c58d4a332d812d15466088da075a18345ffa09daa590018cbb0f791"}
Oct 30 18:50:25.329108 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:25.329087    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:25.330011 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:25.329994    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:50:25.439852 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:25.439799    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv" podStartSLOduration=2.439785339 podStartE2EDuration="2.439785339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:25.438308674 +0000 UTC m=+826.715359119" watchObservedRunningTime="2026-10-30 18:50:25.439785339 +0000 UTC m=+826.716835784"
Oct 30 18:50:25.616378 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:25.616343    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:25.665574 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:25.665524    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w" podStartSLOduration=2.665506756 podStartE2EDuration="2.665506756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:25.661630802 +0000 UTC m=+826.938681247" watchObservedRunningTime="2026-10-30 18:50:25.665506756 +0000 UTC m=+826.942557201"
Oct 30 18:50:25.917180 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:25.917095    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:25.922231 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:25.922199    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:26.339662 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:26.339567    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:26.340518 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:26.340499    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:50:26.494543 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:26.494511    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:26.621165 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:26.621137    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:26.705320 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:26.705295    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:27.278915 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:27.278885    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:27.344009 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:27.343973    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:27.345129 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:27.345056    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:50:27.475903 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:27.475874    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:27.776129 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:27.776099    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:28.018097 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:28.018045    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:28.285551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:28.285513    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:28.681599 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:28.681564    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:28.876877 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:28.876846    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:30.292615 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:30.292581    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:30.498828 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:30.498796    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:30.717614 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:30.717580    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:31.091554 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:31.091477    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:31.311267 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:31.311235    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:31.334744 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:31.334714    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc"]
Oct 30 18:50:31.337888 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:31.337867    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc"
Oct 30 18:50:31.353445 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:31.353425    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8m44\" (UniqueName: \"kubernetes.io/projected/0b676034-384c-4c43-a8f5-991977f109ce-kube-api-access-p8m44\") pod \"payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc\" (UID: \"0b676034-384c-4c43-a8f5-991977f109ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc"
Oct 30 18:50:31.353530 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:31.353486    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b676034-384c-4c43-a8f5-991977f109ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc\" (UID: \"0b676034-384c-4c43-a8f5-991977f109ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc"
Oct 30 18:50:31.391562 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:31.391537    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-649d16\""
Oct 30 18:50:31.434483 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:31.434460    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-649d16-dockercfg-g2p77\""
Oct 30 18:50:31.436163 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:31.436141    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc"]
Oct 30 18:50:31.454606 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:31.454582    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8m44\" (UniqueName: \"kubernetes.io/projected/0b676034-384c-4c43-a8f5-991977f109ce-kube-api-access-p8m44\") pod \"payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc\" (UID: \"0b676034-384c-4c43-a8f5-991977f109ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc"
Oct 30 18:50:31.454697 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:31.454611    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b676034-384c-4c43-a8f5-991977f109ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc\" (UID: \"0b676034-384c-4c43-a8f5-991977f109ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc"
Oct 30 18:50:31.455455 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:31.455438    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b676034-384c-4c43-a8f5-991977f109ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc\" (UID: \"0b676034-384c-4c43-a8f5-991977f109ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc"
Oct 30 18:50:31.509213 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:31.509172    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8m44\" (UniqueName: \"kubernetes.io/projected/0b676034-384c-4c43-a8f5-991977f109ce-kube-api-access-p8m44\") pod \"payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc\" (UID: \"0b676034-384c-4c43-a8f5-991977f109ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc"
Oct 30 18:50:31.651210 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:31.651182    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc"
Oct 30 18:50:32.028489 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:32.028458    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc"]
Oct 30 18:50:32.030956 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:50:32.030927    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b676034_384c_4c43_a8f5_991977f109ce.slice/crio-c1f1446b8d7eeaf49ff8f16bcfc7a2a5de51ec2132a0298f49e68389df5de514 WatchSource:0}: Error finding container c1f1446b8d7eeaf49ff8f16bcfc7a2a5de51ec2132a0298f49e68389df5de514: Status 404 returned error can't find the container with id c1f1446b8d7eeaf49ff8f16bcfc7a2a5de51ec2132a0298f49e68389df5de514
Oct 30 18:50:32.369915 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:32.369882    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc" event={"ID":"0b676034-384c-4c43-a8f5-991977f109ce","Type":"ContainerStarted","Data":"83824a1dbc82e10702511263b7f2b315f237370d8490afc2746a66aa7a51ffb2"}
Oct 30 18:50:32.370569 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:32.369933    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc" event={"ID":"0b676034-384c-4c43-a8f5-991977f109ce","Type":"ContainerStarted","Data":"c1f1446b8d7eeaf49ff8f16bcfc7a2a5de51ec2132a0298f49e68389df5de514"}
Oct 30 18:50:32.370569 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:32.369996    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc"
Oct 30 18:50:32.481942 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:32.481896    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc" podStartSLOduration=1.481881579 podStartE2EDuration="1.481881579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:32.479415168 +0000 UTC m=+833.756465615" watchObservedRunningTime="2026-10-30 18:50:32.481881579 +0000 UTC m=+833.758932055"
Oct 30 18:50:32.855268 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:32.855153    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:33.063406 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:33.063370    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:33.258424 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:33.258337    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:33.473526 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:33.473493    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:33.685370 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:33.685327    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:35.385037 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:35.385000    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:35.603053 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:35.603021    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:35.757324 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:35.757252    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb"]
Oct 30 18:50:35.759705 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:35.759691    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb"
Oct 30 18:50:35.791741 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:35.791711    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w54r8\" (UniqueName: \"kubernetes.io/projected/9918ebe6-aa19-488e-b259-e6201f326650-kube-api-access-w54r8\") pod \"payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb\" (UID: \"9918ebe6-aa19-488e-b259-e6201f326650\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb"
Oct 30 18:50:35.791861 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:35.791769    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9918ebe6-aa19-488e-b259-e6201f326650-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb\" (UID: \"9918ebe6-aa19-488e-b259-e6201f326650\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb"
Oct 30 18:50:35.855445 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:35.855396    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-5f80ac\""
Oct 30 18:50:35.855692 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:35.855516    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-5f80ac-dockercfg-h58rn\""
Oct 30 18:50:35.858056 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:35.858031    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb"]
Oct 30 18:50:35.893161 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:35.893130    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w54r8\" (UniqueName: \"kubernetes.io/projected/9918ebe6-aa19-488e-b259-e6201f326650-kube-api-access-w54r8\") pod \"payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb\" (UID: \"9918ebe6-aa19-488e-b259-e6201f326650\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb"
Oct 30 18:50:35.893288 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:35.893189    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9918ebe6-aa19-488e-b259-e6201f326650-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb\" (UID: \"9918ebe6-aa19-488e-b259-e6201f326650\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb"
Oct 30 18:50:35.893926 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:35.893907    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9918ebe6-aa19-488e-b259-e6201f326650-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb\" (UID: \"9918ebe6-aa19-488e-b259-e6201f326650\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb"
Oct 30 18:50:35.946825 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:35.946795    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6"]
Oct 30 18:50:35.949284 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:35.949263    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6"
Oct 30 18:50:35.952076 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:35.952039    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w54r8\" (UniqueName: \"kubernetes.io/projected/9918ebe6-aa19-488e-b259-e6201f326650-kube-api-access-w54r8\") pod \"payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb\" (UID: \"9918ebe6-aa19-488e-b259-e6201f326650\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb"
Oct 30 18:50:35.994135 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:35.994107    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g67jv\" (UniqueName: \"kubernetes.io/projected/e9720622-6440-4232-8855-ae46ec5bf434-kube-api-access-g67jv\") pod \"payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6\" (UID: \"e9720622-6440-4232-8855-ae46ec5bf434\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6"
Oct 30 18:50:35.994221 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:35.994187    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9720622-6440-4232-8855-ae46ec5bf434-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6\" (UID: \"e9720622-6440-4232-8855-ae46ec5bf434\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6"
Oct 30 18:50:36.005152 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:36.005130    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6"]
Oct 30 18:50:36.071301 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:36.071226    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb"
Oct 30 18:50:36.094870 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:36.094842    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g67jv\" (UniqueName: \"kubernetes.io/projected/e9720622-6440-4232-8855-ae46ec5bf434-kube-api-access-g67jv\") pod \"payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6\" (UID: \"e9720622-6440-4232-8855-ae46ec5bf434\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6"
Oct 30 18:50:36.095020 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:36.094935    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9720622-6440-4232-8855-ae46ec5bf434-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6\" (UID: \"e9720622-6440-4232-8855-ae46ec5bf434\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6"
Oct 30 18:50:36.095645 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:36.095620    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9720622-6440-4232-8855-ae46ec5bf434-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6\" (UID: \"e9720622-6440-4232-8855-ae46ec5bf434\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6"
Oct 30 18:50:36.150754 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:36.150725    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g67jv\" (UniqueName: \"kubernetes.io/projected/e9720622-6440-4232-8855-ae46ec5bf434-kube-api-access-g67jv\") pod \"payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6\" (UID: \"e9720622-6440-4232-8855-ae46ec5bf434\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6"
Oct 30 18:50:36.270015 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:36.269978    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6"
Oct 30 18:50:36.433980 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:36.433956    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb"]
Oct 30 18:50:36.435862 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:50:36.435840    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9918ebe6_aa19_488e_b259_e6201f326650.slice/crio-5a1f856a76ff4cb277d6e0eff14159660b38996332f1ff9e63521ed7fb2eeeed WatchSource:0}: Error finding container 5a1f856a76ff4cb277d6e0eff14159660b38996332f1ff9e63521ed7fb2eeeed: Status 404 returned error can't find the container with id 5a1f856a76ff4cb277d6e0eff14159660b38996332f1ff9e63521ed7fb2eeeed
Oct 30 18:50:36.635194 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:36.635162    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6"]
Oct 30 18:50:36.637363 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:50:36.637323    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9720622_6440_4232_8855_ae46ec5bf434.slice/crio-ecd2b8fa8dfa045f71bbb37f1141717bcaaff30fd588e442b4f44cbf0fe1cc2d WatchSource:0}: Error finding container ecd2b8fa8dfa045f71bbb37f1141717bcaaff30fd588e442b4f44cbf0fe1cc2d: Status 404 returned error can't find the container with id ecd2b8fa8dfa045f71bbb37f1141717bcaaff30fd588e442b4f44cbf0fe1cc2d
Oct 30 18:50:37.394876 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:37.394839    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6" event={"ID":"e9720622-6440-4232-8855-ae46ec5bf434","Type":"ContainerStarted","Data":"a5aefc16cb3c8a3ec160196d15b320d3c3540f6ea9f9e35e6992355eb605d3ab"}
Oct 30 18:50:37.395102 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:37.394890    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6" event={"ID":"e9720622-6440-4232-8855-ae46ec5bf434","Type":"ContainerStarted","Data":"ecd2b8fa8dfa045f71bbb37f1141717bcaaff30fd588e442b4f44cbf0fe1cc2d"}
Oct 30 18:50:37.395102 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:37.394944    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6"
Oct 30 18:50:37.396504 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:37.396478    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb" event={"ID":"9918ebe6-aa19-488e-b259-e6201f326650","Type":"ContainerStarted","Data":"7ecc1ae5bfa9ec792e152a062207ab7d74b9b753618c3c9f21d203482b768de3"}
Oct 30 18:50:37.396628 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:37.396526    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb" event={"ID":"9918ebe6-aa19-488e-b259-e6201f326650","Type":"ContainerStarted","Data":"5a1f856a76ff4cb277d6e0eff14159660b38996332f1ff9e63521ed7fb2eeeed"}
Oct 30 18:50:37.396628 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:37.396554    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb"
Oct 30 18:50:37.505531 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:37.505475    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6" podStartSLOduration=2.505455276 podStartE2EDuration="2.505455276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:37.504258703 +0000 UTC m=+838.781309148" watchObservedRunningTime="2026-10-30 18:50:37.505455276 +0000 UTC m=+838.782505723"
Oct 30 18:50:37.617209 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:37.617150    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb" podStartSLOduration=2.617131778 podStartE2EDuration="2.617131778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:37.616715393 +0000 UTC m=+838.893765837" watchObservedRunningTime="2026-10-30 18:50:37.617131778 +0000 UTC m=+838.894182224"
Oct 30 18:50:37.715377 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:37.715270    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:37.899405 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:37.899375    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:38.073290 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:38.073219    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf"]
Oct 30 18:50:38.076258 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:38.076236    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf"
Oct 30 18:50:38.095542 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:38.095522    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:38.115483 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:38.115457    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j84j7\" (UniqueName: \"kubernetes.io/projected/dc3baffc-49c4-4101-9bf4-047229d89000-kube-api-access-j84j7\") pod \"payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf\" (UID: \"dc3baffc-49c4-4101-9bf4-047229d89000\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf"
Oct 30 18:50:38.115583 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:38.115566    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3baffc-49c4-4101-9bf4-047229d89000-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf\" (UID: \"dc3baffc-49c4-4101-9bf4-047229d89000\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf"
Oct 30 18:50:38.125051 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:38.125010    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-ec8dab\""
Oct 30 18:50:38.125390 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:38.125213    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-ec8dab-dockercfg-5xbls\""
Oct 30 18:50:38.132405 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:38.132379    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf"]
Oct 30 18:50:38.216120 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:38.216060    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j84j7\" (UniqueName: \"kubernetes.io/projected/dc3baffc-49c4-4101-9bf4-047229d89000-kube-api-access-j84j7\") pod \"payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf\" (UID: \"dc3baffc-49c4-4101-9bf4-047229d89000\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf"
Oct 30 18:50:38.216297 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:38.216149    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3baffc-49c4-4101-9bf4-047229d89000-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf\" (UID: \"dc3baffc-49c4-4101-9bf4-047229d89000\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf"
Oct 30 18:50:38.216796 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:38.216775    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3baffc-49c4-4101-9bf4-047229d89000-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf\" (UID: \"dc3baffc-49c4-4101-9bf4-047229d89000\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf"
Oct 30 18:50:38.240366 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:38.240332    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d"]
Oct 30 18:50:38.242930 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:38.242907    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d"
Oct 30 18:50:38.270204 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:38.270175    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j84j7\" (UniqueName: \"kubernetes.io/projected/dc3baffc-49c4-4101-9bf4-047229d89000-kube-api-access-j84j7\") pod \"payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf\" (UID: \"dc3baffc-49c4-4101-9bf4-047229d89000\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf"
Oct 30 18:50:38.289472 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:38.289430    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:38.297115 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:38.297087    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d"]
Oct 30 18:50:38.317498 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:38.317470    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9cd1a58-ecd4-4611-aaac-28109cd2981a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d\" (UID: \"b9cd1a58-ecd4-4611-aaac-28109cd2981a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d"
Oct 30 18:50:38.317653 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:38.317519    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs7mk\" (UniqueName: \"kubernetes.io/projected/b9cd1a58-ecd4-4611-aaac-28109cd2981a-kube-api-access-xs7mk\") pod \"payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d\" (UID: \"b9cd1a58-ecd4-4611-aaac-28109cd2981a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d"
Oct 30 18:50:38.387917 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:38.387889    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf"
Oct 30 18:50:38.418162 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:38.418133    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs7mk\" (UniqueName: \"kubernetes.io/projected/b9cd1a58-ecd4-4611-aaac-28109cd2981a-kube-api-access-xs7mk\") pod \"payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d\" (UID: \"b9cd1a58-ecd4-4611-aaac-28109cd2981a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d"
Oct 30 18:50:38.418317 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:38.418272    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9cd1a58-ecd4-4611-aaac-28109cd2981a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d\" (UID: \"b9cd1a58-ecd4-4611-aaac-28109cd2981a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d"
Oct 30 18:50:38.418966 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:38.418942    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9cd1a58-ecd4-4611-aaac-28109cd2981a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d\" (UID: \"b9cd1a58-ecd4-4611-aaac-28109cd2981a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d"
Oct 30 18:50:38.473107 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:38.473082    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs7mk\" (UniqueName: \"kubernetes.io/projected/b9cd1a58-ecd4-4611-aaac-28109cd2981a-kube-api-access-xs7mk\") pod \"payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d\" (UID: \"b9cd1a58-ecd4-4611-aaac-28109cd2981a\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d"
Oct 30 18:50:38.560166 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:38.560133    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d"
Oct 30 18:50:38.752896 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:38.752868    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf"]
Oct 30 18:50:38.755114 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:50:38.755084    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3baffc_49c4_4101_9bf4_047229d89000.slice/crio-b0996816d7d5b5bcc729cf9b0ab7f749fc9a889324348f968452269ad9eeb238 WatchSource:0}: Error finding container b0996816d7d5b5bcc729cf9b0ab7f749fc9a889324348f968452269ad9eeb238: Status 404 returned error can't find the container with id b0996816d7d5b5bcc729cf9b0ab7f749fc9a889324348f968452269ad9eeb238
Oct 30 18:50:38.922370 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:38.922335    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d"]
Oct 30 18:50:39.408336 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:39.408300    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf" event={"ID":"dc3baffc-49c4-4101-9bf4-047229d89000","Type":"ContainerStarted","Data":"922c79e8290ed4f53737c6383a853630095dcff3ea014c272638e4fdece41ef1"}
Oct 30 18:50:39.408513 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:39.408392    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf" event={"ID":"dc3baffc-49c4-4101-9bf4-047229d89000","Type":"ContainerStarted","Data":"b0996816d7d5b5bcc729cf9b0ab7f749fc9a889324348f968452269ad9eeb238"}
Oct 30 18:50:39.408513 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:39.408422    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf"
Oct 30 18:50:39.409870 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:39.409842    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d" event={"ID":"b9cd1a58-ecd4-4611-aaac-28109cd2981a","Type":"ContainerStarted","Data":"2164deca5b6f9cbd971790debf5d890e560ef56d52597679275771209c898bc1"}
Oct 30 18:50:39.410007 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:39.409882    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d" event={"ID":"b9cd1a58-ecd4-4611-aaac-28109cd2981a","Type":"ContainerStarted","Data":"4a295b0d3e01c320b372484224f5d85e962c642c36f128c5f0e0b5ca4799d47f"}
Oct 30 18:50:39.410007 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:39.409996    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d"
Oct 30 18:50:39.526915 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:39.526861    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf" podStartSLOduration=1.526848283 podStartE2EDuration="1.526848283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:39.524037852 +0000 UTC m=+840.801088297" watchObservedRunningTime="2026-10-30 18:50:39.526848283 +0000 UTC m=+840.803898751"
Oct 30 18:50:39.632814 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:39.632762    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d" podStartSLOduration=1.6327479550000001 podStartE2EDuration="1.632747955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:39.631315551 +0000 UTC m=+840.908365996" watchObservedRunningTime="2026-10-30 18:50:39.632747955 +0000 UTC m=+840.909798400"
Oct 30 18:50:40.128624 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:40.128593    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:40.310347 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:40.310316    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:40.615996 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:40.615958    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94"]
Oct 30 18:50:40.618770 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:40.618748    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94"
Oct 30 18:50:40.668462 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:40.668430    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f5705f\""
Oct 30 18:50:40.671273 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:40.671242    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jbn9\" (UniqueName: \"kubernetes.io/projected/21e5d49b-a1e4-4232-b823-1d35a77b1ab3-kube-api-access-9jbn9\") pod \"payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94\" (UID: \"21e5d49b-a1e4-4232-b823-1d35a77b1ab3\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94"
Oct 30 18:50:40.671437 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:40.671371    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e5d49b-a1e4-4232-b823-1d35a77b1ab3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94\" (UID: \"21e5d49b-a1e4-4232-b823-1d35a77b1ab3\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94"
Oct 30 18:50:40.714933 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:40.714899    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-f5705f-dockercfg-qhwqf\""
Oct 30 18:50:40.717141 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:40.717119    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94"]
Oct 30 18:50:40.772453 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:40.772418    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e5d49b-a1e4-4232-b823-1d35a77b1ab3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94\" (UID: \"21e5d49b-a1e4-4232-b823-1d35a77b1ab3\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94"
Oct 30 18:50:40.772648 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:40.772504    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jbn9\" (UniqueName: \"kubernetes.io/projected/21e5d49b-a1e4-4232-b823-1d35a77b1ab3-kube-api-access-9jbn9\") pod \"payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94\" (UID: \"21e5d49b-a1e4-4232-b823-1d35a77b1ab3\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94"
Oct 30 18:50:40.773214 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:40.773191    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e5d49b-a1e4-4232-b823-1d35a77b1ab3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94\" (UID: \"21e5d49b-a1e4-4232-b823-1d35a77b1ab3\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94"
Oct 30 18:50:40.827553 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:40.827519    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jbn9\" (UniqueName: \"kubernetes.io/projected/21e5d49b-a1e4-4232-b823-1d35a77b1ab3-kube-api-access-9jbn9\") pod \"payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94\" (UID: \"21e5d49b-a1e4-4232-b823-1d35a77b1ab3\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94"
Oct 30 18:50:40.931203 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:40.931118    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94"
Oct 30 18:50:41.295297 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:41.295270    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94"]
Oct 30 18:50:41.297630 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:50:41.297588    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e5d49b_a1e4_4232_b823_1d35a77b1ab3.slice/crio-2b95dae47c647476ed0acb7a99467bd2a626e66b87e5ee44c663cf323aaa987f WatchSource:0}: Error finding container 2b95dae47c647476ed0acb7a99467bd2a626e66b87e5ee44c663cf323aaa987f: Status 404 returned error can't find the container with id 2b95dae47c647476ed0acb7a99467bd2a626e66b87e5ee44c663cf323aaa987f
Oct 30 18:50:41.421897 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:41.421864    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94" event={"ID":"21e5d49b-a1e4-4232-b823-1d35a77b1ab3","Type":"ContainerStarted","Data":"b681b8d2e7afe4fc4032c08750b3ee91e0058c36c4731839ad5fd22198ee55bf"}
Oct 30 18:50:41.422049 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:41.421920    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94" event={"ID":"21e5d49b-a1e4-4232-b823-1d35a77b1ab3","Type":"ContainerStarted","Data":"2b95dae47c647476ed0acb7a99467bd2a626e66b87e5ee44c663cf323aaa987f"}
Oct 30 18:50:41.422049 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:41.422031    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94"
Oct 30 18:50:41.530905 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:41.530858    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94" podStartSLOduration=1.530844098 podStartE2EDuration="1.530844098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:41.52939875 +0000 UTC m=+842.806449191" watchObservedRunningTime="2026-10-30 18:50:41.530844098 +0000 UTC m=+842.807894542"
Oct 30 18:50:42.475837 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:42.475803    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:42.666993 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:42.666948    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:42.981890 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:42.981861    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs"]
Oct 30 18:50:42.984440 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:42.984419    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs"
Oct 30 18:50:43.034934 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:43.034908    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-a1617e\""
Oct 30 18:50:43.081696 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:43.081667    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-a1617e-dockercfg-7qg7z\""
Oct 30 18:50:43.083594 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:43.083575    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs"]
Oct 30 18:50:43.094794 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:43.094769    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nmr7\" (UniqueName: \"kubernetes.io/projected/156d3474-d19a-4022-9a3c-88b93e53b527-kube-api-access-6nmr7\") pod \"payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs\" (UID: \"156d3474-d19a-4022-9a3c-88b93e53b527\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs"
Oct 30 18:50:43.094914 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:43.094831    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156d3474-d19a-4022-9a3c-88b93e53b527-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs\" (UID: \"156d3474-d19a-4022-9a3c-88b93e53b527\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs"
Oct 30 18:50:43.195919 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:43.195892    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156d3474-d19a-4022-9a3c-88b93e53b527-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs\" (UID: \"156d3474-d19a-4022-9a3c-88b93e53b527\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs"
Oct 30 18:50:43.196095 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:43.195980    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nmr7\" (UniqueName: \"kubernetes.io/projected/156d3474-d19a-4022-9a3c-88b93e53b527-kube-api-access-6nmr7\") pod \"payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs\" (UID: \"156d3474-d19a-4022-9a3c-88b93e53b527\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs"
Oct 30 18:50:43.196596 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:43.196576    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156d3474-d19a-4022-9a3c-88b93e53b527-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs\" (UID: \"156d3474-d19a-4022-9a3c-88b93e53b527\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs"
Oct 30 18:50:43.253144 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:43.253084    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nmr7\" (UniqueName: \"kubernetes.io/projected/156d3474-d19a-4022-9a3c-88b93e53b527-kube-api-access-6nmr7\") pod \"payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs\" (UID: \"156d3474-d19a-4022-9a3c-88b93e53b527\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs"
Oct 30 18:50:43.296171 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:43.296142    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs"
Oct 30 18:50:43.378766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:43.378741    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc"
Oct 30 18:50:43.379661 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:43.379635    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:43.611280 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:43.611250    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:43.671918 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:43.671823    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs"]
Oct 30 18:50:43.674373 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:50:43.674339    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156d3474_d19a_4022_9a3c_88b93e53b527.slice/crio-c07c8bcf98f42393999accf69069e5bd9a7fcd143e5c95324cb4869ffc6f5109 WatchSource:0}: Error finding container c07c8bcf98f42393999accf69069e5bd9a7fcd143e5c95324cb4869ffc6f5109: Status 404 returned error can't find the container with id c07c8bcf98f42393999accf69069e5bd9a7fcd143e5c95324cb4869ffc6f5109
Oct 30 18:50:43.822962 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:43.822929    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:44.437921 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:44.437888    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs" event={"ID":"156d3474-d19a-4022-9a3c-88b93e53b527","Type":"ContainerStarted","Data":"a738ae0497f8e4bd70d0d7068ee0873ba6e2c610f42eb86962fd63ef69bbbbdc"}
Oct 30 18:50:44.438122 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:44.437941    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs" event={"ID":"156d3474-d19a-4022-9a3c-88b93e53b527","Type":"ContainerStarted","Data":"c07c8bcf98f42393999accf69069e5bd9a7fcd143e5c95324cb4869ffc6f5109"}
Oct 30 18:50:44.438122 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:44.437964    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs"
Oct 30 18:50:44.547163 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:44.547110    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs" podStartSLOduration=2.547096172 podStartE2EDuration="2.547096172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:44.546166478 +0000 UTC m=+845.823216923" watchObservedRunningTime="2026-10-30 18:50:44.547096172 +0000 UTC m=+845.824146620"
Oct 30 18:50:45.710669 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:50:45.710636    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:50:47.072536 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:47.072497    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:47.295047 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:47.295010    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:47.449861 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:47.449831    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq"]
Oct 30 18:50:47.453312 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:47.453292    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq"
Oct 30 18:50:47.509600 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:47.509568    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq"]
Oct 30 18:50:47.540563 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:47.540527    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qccdh\" (UniqueName: \"kubernetes.io/projected/539e8daa-2f24-4d43-8017-237fb777d90e-kube-api-access-qccdh\") pod \"payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq\" (UID: \"539e8daa-2f24-4d43-8017-237fb777d90e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq"
Oct 30 18:50:47.540727 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:47.540568    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/539e8daa-2f24-4d43-8017-237fb777d90e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq\" (UID: \"539e8daa-2f24-4d43-8017-237fb777d90e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq"
Oct 30 18:50:47.552661 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:47.552639    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-3b6ad6\""
Oct 30 18:50:47.552864 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:47.552844    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-3b6ad6-dockercfg-7lflg\""
Oct 30 18:50:47.641217 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:47.641182    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qccdh\" (UniqueName: \"kubernetes.io/projected/539e8daa-2f24-4d43-8017-237fb777d90e-kube-api-access-qccdh\") pod \"payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq\" (UID: \"539e8daa-2f24-4d43-8017-237fb777d90e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq"
Oct 30 18:50:47.641217 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:47.641220    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/539e8daa-2f24-4d43-8017-237fb777d90e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq\" (UID: \"539e8daa-2f24-4d43-8017-237fb777d90e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq"
Oct 30 18:50:47.641955 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:47.641934    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/539e8daa-2f24-4d43-8017-237fb777d90e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq\" (UID: \"539e8daa-2f24-4d43-8017-237fb777d90e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq"
Oct 30 18:50:47.696886 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:47.696858    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qccdh\" (UniqueName: \"kubernetes.io/projected/539e8daa-2f24-4d43-8017-237fb777d90e-kube-api-access-qccdh\") pod \"payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq\" (UID: \"539e8daa-2f24-4d43-8017-237fb777d90e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq"
Oct 30 18:50:47.765839 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:47.765763    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq"
Oct 30 18:50:48.132362 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:48.132337    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq"]
Oct 30 18:50:48.134528 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:50:48.134495    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod539e8daa_2f24_4d43_8017_237fb777d90e.slice/crio-1efda1fdef41aff662e850beee251c2c2096591b3f9935ab03014ee16f9b038c WatchSource:0}: Error finding container 1efda1fdef41aff662e850beee251c2c2096591b3f9935ab03014ee16f9b038c: Status 404 returned error can't find the container with id 1efda1fdef41aff662e850beee251c2c2096591b3f9935ab03014ee16f9b038c
Oct 30 18:50:48.405107 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:48.405009    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6"
Oct 30 18:50:48.405107 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:48.405083    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb"
Oct 30 18:50:48.458944 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:48.458918    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq" event={"ID":"539e8daa-2f24-4d43-8017-237fb777d90e","Type":"ContainerStarted","Data":"d8cbacada5c40930e6550b79d44fc1752c4ec5f2df7884738014c20e0cc0eb04"}
Oct 30 18:50:48.459094 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:48.458964    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq" event={"ID":"539e8daa-2f24-4d43-8017-237fb777d90e","Type":"ContainerStarted","Data":"1efda1fdef41aff662e850beee251c2c2096591b3f9935ab03014ee16f9b038c"}
Oct 30 18:50:48.459094 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:48.458997    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq"
Oct 30 18:50:48.734001 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:48.733901    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq" podStartSLOduration=1.7338860440000001 podStartE2EDuration="1.733886044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:48.732310273 +0000 UTC m=+850.009360718" watchObservedRunningTime="2026-10-30 18:50:48.733886044 +0000 UTC m=+850.010936488"
Oct 30 18:50:48.788294 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:48.788263    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:48.980924 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:48.980865    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:49.168459 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:49.168428    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:50.427960 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:50.427876    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf"
Oct 30 18:50:50.428474 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:50.428043    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d"
Oct 30 18:50:51.727870 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:51.727837    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:51.927914 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:51.927875    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:52.076502 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:52.076424    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4"]
Oct 30 18:50:52.079766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:52.079745    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4"
Oct 30 18:50:52.175868 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:52.175832    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-75e91b\""
Oct 30 18:50:52.176049 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:52.175928    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-75e91b-dockercfg-cpsnt\""
Oct 30 18:50:52.179750 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:52.179720    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4"]
Oct 30 18:50:52.188427 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:52.188391    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3dd0455-39e6-4b1b-80ee-ffbde02d4f00-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4\" (UID: \"d3dd0455-39e6-4b1b-80ee-ffbde02d4f00\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4"
Oct 30 18:50:52.188643 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:52.188621    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd8qx\" (UniqueName: \"kubernetes.io/projected/d3dd0455-39e6-4b1b-80ee-ffbde02d4f00-kube-api-access-xd8qx\") pod \"payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4\" (UID: \"d3dd0455-39e6-4b1b-80ee-ffbde02d4f00\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4"
Oct 30 18:50:52.229831 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:52.229783    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c"]
Oct 30 18:50:52.235122 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:52.235094    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c"
Oct 30 18:50:52.275873 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:52.275840    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c"]
Oct 30 18:50:52.290123 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:52.290059    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3dd0455-39e6-4b1b-80ee-ffbde02d4f00-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4\" (UID: \"d3dd0455-39e6-4b1b-80ee-ffbde02d4f00\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4"
Oct 30 18:50:52.290297 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:52.290197    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd8qx\" (UniqueName: \"kubernetes.io/projected/d3dd0455-39e6-4b1b-80ee-ffbde02d4f00-kube-api-access-xd8qx\") pod \"payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4\" (UID: \"d3dd0455-39e6-4b1b-80ee-ffbde02d4f00\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4"
Oct 30 18:50:52.290818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:52.290797    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3dd0455-39e6-4b1b-80ee-ffbde02d4f00-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4\" (UID: \"d3dd0455-39e6-4b1b-80ee-ffbde02d4f00\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4"
Oct 30 18:50:52.347511 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:52.347488    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd8qx\" (UniqueName: \"kubernetes.io/projected/d3dd0455-39e6-4b1b-80ee-ffbde02d4f00-kube-api-access-xd8qx\") pod \"payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4\" (UID: \"d3dd0455-39e6-4b1b-80ee-ffbde02d4f00\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4"
Oct 30 18:50:52.390822 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:52.390783    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0580aebc-4e9e-4578-872c-943719ee559f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c\" (UID: \"0580aebc-4e9e-4578-872c-943719ee559f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c"
Oct 30 18:50:52.390982 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:52.390959    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmtcd\" (UniqueName: \"kubernetes.io/projected/0580aebc-4e9e-4578-872c-943719ee559f-kube-api-access-gmtcd\") pod \"payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c\" (UID: \"0580aebc-4e9e-4578-872c-943719ee559f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c"
Oct 30 18:50:52.392623 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:52.392602    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4"
Oct 30 18:50:52.435333 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:52.435309    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94"
Oct 30 18:50:52.493612 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:52.493582    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmtcd\" (UniqueName: \"kubernetes.io/projected/0580aebc-4e9e-4578-872c-943719ee559f-kube-api-access-gmtcd\") pod \"payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c\" (UID: \"0580aebc-4e9e-4578-872c-943719ee559f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c"
Oct 30 18:50:52.493747 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:52.493656    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0580aebc-4e9e-4578-872c-943719ee559f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c\" (UID: \"0580aebc-4e9e-4578-872c-943719ee559f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c"
Oct 30 18:50:52.494640 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:52.494615    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0580aebc-4e9e-4578-872c-943719ee559f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c\" (UID: \"0580aebc-4e9e-4578-872c-943719ee559f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c"
Oct 30 18:50:52.570451 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:52.570423    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmtcd\" (UniqueName: \"kubernetes.io/projected/0580aebc-4e9e-4578-872c-943719ee559f-kube-api-access-gmtcd\") pod \"payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c\" (UID: \"0580aebc-4e9e-4578-872c-943719ee559f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c"
Oct 30 18:50:52.795136 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:52.795112    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4"]
Oct 30 18:50:52.797298 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:50:52.797273    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3dd0455_39e6_4b1b_80ee_ffbde02d4f00.slice/crio-aa47952eeab34e5e76b599b894c1c6d7bf08a5480f9396a89c594183b9cf9d47 WatchSource:0}: Error finding container aa47952eeab34e5e76b599b894c1c6d7bf08a5480f9396a89c594183b9cf9d47: Status 404 returned error can't find the container with id aa47952eeab34e5e76b599b894c1c6d7bf08a5480f9396a89c594183b9cf9d47
Oct 30 18:50:52.848734 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:52.848710    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c"
Oct 30 18:50:53.245206 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:53.245179    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c"]
Oct 30 18:50:53.247484 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:50:53.247450    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0580aebc_4e9e_4578_872c_943719ee559f.slice/crio-c53bf04532a0ff7b0ca80117dcb68f5e5baaf2b6354ed4659417ad2ff150ef95 WatchSource:0}: Error finding container c53bf04532a0ff7b0ca80117dcb68f5e5baaf2b6354ed4659417ad2ff150ef95: Status 404 returned error can't find the container with id c53bf04532a0ff7b0ca80117dcb68f5e5baaf2b6354ed4659417ad2ff150ef95
Oct 30 18:50:53.486372 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:53.486332    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c" event={"ID":"0580aebc-4e9e-4578-872c-943719ee559f","Type":"ContainerStarted","Data":"5e4b153de2dc19cbf358ec5addca655afdce7fe5f1471a8003e66065b16c1282"}
Oct 30 18:50:53.486543 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:53.486397    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c" event={"ID":"0580aebc-4e9e-4578-872c-943719ee559f","Type":"ContainerStarted","Data":"c53bf04532a0ff7b0ca80117dcb68f5e5baaf2b6354ed4659417ad2ff150ef95"}
Oct 30 18:50:53.486543 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:53.486521    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c"
Oct 30 18:50:53.488030 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:53.488007    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4" event={"ID":"d3dd0455-39e6-4b1b-80ee-ffbde02d4f00","Type":"ContainerStarted","Data":"389d884c1ef4feb0839e774bb9640ea12d9d699110005bcda5dda8022a41cf29"}
Oct 30 18:50:53.488137 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:53.488046    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4" event={"ID":"d3dd0455-39e6-4b1b-80ee-ffbde02d4f00","Type":"ContainerStarted","Data":"aa47952eeab34e5e76b599b894c1c6d7bf08a5480f9396a89c594183b9cf9d47"}
Oct 30 18:50:53.488137 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:53.488101    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4"
Oct 30 18:50:53.595947 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:53.595856    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c" podStartSLOduration=1.595841176 podStartE2EDuration="1.595841176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:53.595511105 +0000 UTC m=+854.872561550" watchObservedRunningTime="2026-10-30 18:50:53.595841176 +0000 UTC m=+854.872891620"
Oct 30 18:50:53.730086 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:53.722528    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4" podStartSLOduration=1.722510447 podStartE2EDuration="1.722510447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:53.721792022 +0000 UTC m=+854.998842469" watchObservedRunningTime="2026-10-30 18:50:53.722510447 +0000 UTC m=+854.999560890"
Oct 30 18:50:53.976613 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:53.976581    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:55.446921 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:55.446893    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs"
Oct 30 18:50:57.384245 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:57.384204    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 18:50:57.384656 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:57.384270    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv" podUID="ccddacc3-f62d-477a-9c8b-9ead25b237c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:50:59.003822 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:59.003472    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:59.219153 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:59.218258    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:50:59.467017 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:59.466986    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq"
Oct 30 18:50:59.545494 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:59.545409    4924 generic.go:356] "Generic (PLEG): container finished" podID="ccddacc3-f62d-477a-9c8b-9ead25b237c4" containerID="24edc2c8cc0a4bd1289895d4ca56ccc2dfe7b7f1310385b5f6bea38b7d9b1871" exitCode=0
Oct 30 18:50:59.545494 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:59.545483    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv" event={"ID":"ccddacc3-f62d-477a-9c8b-9ead25b237c4","Type":"ContainerDied","Data":"24edc2c8cc0a4bd1289895d4ca56ccc2dfe7b7f1310385b5f6bea38b7d9b1871"}
Oct 30 18:50:59.545867 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:59.545853    4924 scope.go:122] "RemoveContainer" containerID="24edc2c8cc0a4bd1289895d4ca56ccc2dfe7b7f1310385b5f6bea38b7d9b1871"
Oct 30 18:50:59.547153 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:59.547128    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:50:59.547208 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:59.547198    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:50:59.547243 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:50:59.547233    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:51:00.558031 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:00.557989    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv" event={"ID":"ccddacc3-f62d-477a-9c8b-9ead25b237c4","Type":"ContainerStarted","Data":"000e132108360b6e8a9560dc833ccabdceb7c3409d69cc92d224cb02fd118fb2"}
Oct 30 18:51:00.919088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:00.918475    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:51:00.919088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:00.918606    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 18:51:00.919088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:00.918652    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv" podUID="ccddacc3-f62d-477a-9c8b-9ead25b237c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 18:51:01.923090 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:01.918520    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 18:51:01.923090 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:01.918581    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv" podUID="ccddacc3-f62d-477a-9c8b-9ead25b237c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 18:51:02.923580 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:02.923536    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 18:51:02.924254 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:02.924216    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv" podUID="ccddacc3-f62d-477a-9c8b-9ead25b237c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 18:51:03.923023 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:03.922533    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 18:51:03.923023 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:03.922599    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv" podUID="ccddacc3-f62d-477a-9c8b-9ead25b237c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 18:51:04.075104 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:04.075057    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:04.270028 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:04.269952    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:04.496667 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:04.496639    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4"
Oct 30 18:51:04.496833 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:04.496704    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c"
Oct 30 18:51:04.924441 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:04.924405    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 18:51:04.924713 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:04.924679    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv" podUID="ccddacc3-f62d-477a-9c8b-9ead25b237c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 18:51:05.921041 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:05.921011    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:51:05.921508 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:05.921313    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:51:05.921949 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:05.921931    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 18:51:07.597947 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:07.597915    4924 generic.go:356] "Generic (PLEG): container finished" podID="1e73c307-868e-40e8-ad04-099a7446ccef" containerID="a3e84af7e790444c71aebec9462ee8037428558eaad1a91b9c35353b317a238e" exitCode=0
Oct 30 18:51:07.598390 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:07.597994    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f" event={"ID":"1e73c307-868e-40e8-ad04-099a7446ccef","Type":"ContainerDied","Data":"a3e84af7e790444c71aebec9462ee8037428558eaad1a91b9c35353b317a238e"}
Oct 30 18:51:07.598490 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:07.598467    4924 scope.go:122] "RemoveContainer" containerID="a3e84af7e790444c71aebec9462ee8037428558eaad1a91b9c35353b317a238e"
Oct 30 18:51:07.599624 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:07.599596    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:51:07.599703 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:07.599677    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:51:07.599742 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:07.599709    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:51:08.623089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:08.621719    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f" event={"ID":"1e73c307-868e-40e8-ad04-099a7446ccef","Type":"ContainerStarted","Data":"3652bb640212f6cdd25b148d97e83be90cf657b4972e7f0ae17815306ad84c62"}
Oct 30 18:51:08.950971 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:08.950879    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:51:08.952207 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:08.952181    4924 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 18:51:08.952431 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:08.952404    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f" podUID="1e73c307-868e-40e8-ad04-099a7446ccef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 18:51:09.078778 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:09.078749    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:09.944088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:09.943647    4924 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 18:51:09.944088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:09.943725    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f" podUID="1e73c307-868e-40e8-ad04-099a7446ccef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 18:51:10.329356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:10.329241    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:51:10.947548 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:10.947508    4924 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 18:51:10.948133 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:10.948101    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f" podUID="1e73c307-868e-40e8-ad04-099a7446ccef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 18:51:11.945563 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:11.945525    4924 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 18:51:11.945853 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:11.945822    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f" podUID="1e73c307-868e-40e8-ad04-099a7446ccef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 18:51:12.943737 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:12.943689    4924 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 30 18:51:12.944215 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:12.943774    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f" podUID="1e73c307-868e-40e8-ad04-099a7446ccef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 30 18:51:13.811844 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:13.811797    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:13.948626 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:13.948595    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:51:13.950009 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:13.949984    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 18:51:14.002097 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:14.002021    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:14.190492 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:14.190454    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:15.711196 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:51:15.711157    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:51:16.689361 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:16.689323    4924 generic.go:356] "Generic (PLEG): container finished" podID="f53aa846-ec01-4b44-a518-49a558965906" containerID="b7ed64f2a0ffae646e5fd3f7e0d50c1ce291c58ad8d2678edbae7d7165eaf89a" exitCode=0
Oct 30 18:51:16.689606 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:16.689398    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g" event={"ID":"f53aa846-ec01-4b44-a518-49a558965906","Type":"ContainerDied","Data":"b7ed64f2a0ffae646e5fd3f7e0d50c1ce291c58ad8d2678edbae7d7165eaf89a"}
Oct 30 18:51:16.689855 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:16.689837    4924 scope.go:122] "RemoveContainer" containerID="b7ed64f2a0ffae646e5fd3f7e0d50c1ce291c58ad8d2678edbae7d7165eaf89a"
Oct 30 18:51:16.691129 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:16.691104    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:51:16.691194 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:16.691169    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:51:16.691232 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:16.691198    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:51:17.663094 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:17.663045    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:17.695968 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:17.695934    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g" event={"ID":"f53aa846-ec01-4b44-a518-49a558965906","Type":"ContainerStarted","Data":"01ad1b91e81af2696711f1b54ce954f9b675972822cdd80c2a184acf8a10209b"}
Oct 30 18:51:18.244216 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:18.244188    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:18.286898 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:18.286866    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:51:18.287218 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:18.287085    4924 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 18:51:18.287319 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:18.287257    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g" podUID="f53aa846-ec01-4b44-a518-49a558965906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 18:51:18.430692 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:18.430656    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:19.287105 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:19.287055    4924 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 18:51:19.287614 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:19.287122    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g" podUID="f53aa846-ec01-4b44-a518-49a558965906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 18:51:19.524093 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:19.522142    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:20.287179 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:20.287133    4924 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 18:51:20.287569 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:20.287214    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g" podUID="f53aa846-ec01-4b44-a518-49a558965906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 18:51:21.287302 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.287265    4924 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 18:51:21.287704 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.287336    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g" podUID="f53aa846-ec01-4b44-a518-49a558965906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 18:51:21.441322 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.441287    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"]
Oct 30 18:51:21.444459 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.444432    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:51:21.494276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.494246    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5f80ac\"/\"openshift-service-ca.crt\""
Oct 30 18:51:21.495742 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.495719    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-tmp-dir\") pod \"simulator-w1-5f80ac-kserve-69cdd574df-2vfzd\" (UID: \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\") " pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:51:21.495837 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.495821    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-dshm\") pod \"simulator-w1-5f80ac-kserve-69cdd574df-2vfzd\" (UID: \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\") " pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:51:21.495876 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.495864    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-tls-certs\") pod \"simulator-w1-5f80ac-kserve-69cdd574df-2vfzd\" (UID: \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\") " pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:51:21.495932 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.495917    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnsp8\" (UniqueName: \"kubernetes.io/projected/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-kube-api-access-pnsp8\") pod \"simulator-w1-5f80ac-kserve-69cdd574df-2vfzd\" (UID: \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\") " pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:51:21.495968 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.495954    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-home\") pod \"simulator-w1-5f80ac-kserve-69cdd574df-2vfzd\" (UID: \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\") " pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:51:21.496072 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.496044    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-model-cache\") pod \"simulator-w1-5f80ac-kserve-69cdd574df-2vfzd\" (UID: \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\") " pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:51:21.540949 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.540861    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f80ac\"/\"simulator-w1-5f80ac-kserve-self-signed-certs\""
Oct 30 18:51:21.540949 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.540861    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f80ac\"/\"default-dockercfg-dt6dr\""
Oct 30 18:51:21.540949 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.540876    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5f80ac\"/\"kube-root-ca.crt\""
Oct 30 18:51:21.543148 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.543127    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"]
Oct 30 18:51:21.583646 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.583622    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:21.597133 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.597101    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-dshm\") pod \"simulator-w1-5f80ac-kserve-69cdd574df-2vfzd\" (UID: \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\") " pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:51:21.597361 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.597155    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-tls-certs\") pod \"simulator-w1-5f80ac-kserve-69cdd574df-2vfzd\" (UID: \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\") " pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:51:21.597361 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.597185    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnsp8\" (UniqueName: \"kubernetes.io/projected/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-kube-api-access-pnsp8\") pod \"simulator-w1-5f80ac-kserve-69cdd574df-2vfzd\" (UID: \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\") " pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:51:21.597361 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.597229    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-home\") pod \"simulator-w1-5f80ac-kserve-69cdd574df-2vfzd\" (UID: \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\") " pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:51:21.597361 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.597314    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-model-cache\") pod \"simulator-w1-5f80ac-kserve-69cdd574df-2vfzd\" (UID: \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\") " pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:51:21.597361 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.597353    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-tmp-dir\") pod \"simulator-w1-5f80ac-kserve-69cdd574df-2vfzd\" (UID: \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\") " pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:51:21.597758 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.597734    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-tmp-dir\") pod \"simulator-w1-5f80ac-kserve-69cdd574df-2vfzd\" (UID: \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\") " pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:51:21.597906 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.597808    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-home\") pod \"simulator-w1-5f80ac-kserve-69cdd574df-2vfzd\" (UID: \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\") " pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:51:21.597977 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.597867    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-model-cache\") pod \"simulator-w1-5f80ac-kserve-69cdd574df-2vfzd\" (UID: \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\") " pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:51:21.599802 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.599757    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-dshm\") pod \"simulator-w1-5f80ac-kserve-69cdd574df-2vfzd\" (UID: \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\") " pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:51:21.600338 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.600314    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-tls-certs\") pod \"simulator-w1-5f80ac-kserve-69cdd574df-2vfzd\" (UID: \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\") " pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:51:21.651739 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.651705    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnsp8\" (UniqueName: \"kubernetes.io/projected/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-kube-api-access-pnsp8\") pod \"simulator-w1-5f80ac-kserve-69cdd574df-2vfzd\" (UID: \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\") " pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:51:21.755904 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:21.755875    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:51:22.149249 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:51:22.149218    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03ac954a_d07b_4523_8d5a_a13dffd0bd2e.slice/crio-d0e297abd8251ffd89ba690a4071f7b0a3d36e2b89b94360324e864b6483f820 WatchSource:0}: Error finding container d0e297abd8251ffd89ba690a4071f7b0a3d36e2b89b94360324e864b6483f820: Status 404 returned error can't find the container with id d0e297abd8251ffd89ba690a4071f7b0a3d36e2b89b94360324e864b6483f820
Oct 30 18:51:22.151256 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:22.151233    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"]
Oct 30 18:51:22.287623 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:22.287585    4924 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 30 18:51:22.287980 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:22.287651    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g" podUID="f53aa846-ec01-4b44-a518-49a558965906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 30 18:51:22.757531 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:22.757483    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd" event={"ID":"03ac954a-d07b-4523-8d5a-a13dffd0bd2e","Type":"ContainerStarted","Data":"15805d7ed73fcef47a8b31dbe411a6bf260d0ffa4879407d4b097070f068f360"}
Oct 30 18:51:22.757753 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:22.757543    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd" event={"ID":"03ac954a-d07b-4523-8d5a-a13dffd0bd2e","Type":"ContainerStarted","Data":"d0e297abd8251ffd89ba690a4071f7b0a3d36e2b89b94360324e864b6483f820"}
Oct 30 18:51:22.868874 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:22.868817    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd" podStartSLOduration=1.8688023679999999 podStartE2EDuration="1.868802368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:22.867159745 +0000 UTC m=+884.144210203" watchObservedRunningTime="2026-10-30 18:51:22.868802368 +0000 UTC m=+884.145852812"
Oct 30 18:51:23.291559 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:23.291525    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:51:23.291929 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:23.291807    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:51:23.292416 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:23.292397    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 18:51:23.756580 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:23.756551    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:51:24.242309 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.242273    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"]
Oct 30 18:51:24.244984 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.244960    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.341491 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.341453    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-75e91b-openshift-default-dockercfg-mcrrx\""
Oct 30 18:51:24.344005 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.343977    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"]
Oct 30 18:51:24.348986 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.348956    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dab0473-fa74-460f-8b18-69874157c8fa-istiod-ca-cert\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.349120 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.349020    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-istio-data\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.349170 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.349149    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-credential-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.349218 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.349179    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-istio-envoy\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.349218 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.349200    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-workload-certs\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.349324 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.349242    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dab0473-fa74-460f-8b18-69874157c8fa-istio-token\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.349324 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.349278    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9bjx\" (UniqueName: \"kubernetes.io/projected/4dab0473-fa74-460f-8b18-69874157c8fa-kube-api-access-z9bjx\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.349435 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.349416    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-workload-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.349479 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.349466    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dab0473-fa74-460f-8b18-69874157c8fa-istio-podinfo\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.450611 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.450577    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-workload-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.450806 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.450620    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dab0473-fa74-460f-8b18-69874157c8fa-istio-podinfo\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.450806 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.450660    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dab0473-fa74-460f-8b18-69874157c8fa-istiod-ca-cert\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.450806 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.450696    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-istio-data\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.450806 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.450736    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-credential-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.451028 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.450862    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-istio-envoy\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.451028 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.450906    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-workload-certs\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.451028 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.450933    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dab0473-fa74-460f-8b18-69874157c8fa-istio-token\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.451028 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.450963    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9bjx\" (UniqueName: \"kubernetes.io/projected/4dab0473-fa74-460f-8b18-69874157c8fa-kube-api-access-z9bjx\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.451028 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.451002    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-workload-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.451275 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.451043    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-istio-data\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.451317 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.451276    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-workload-certs\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.451437 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.451417    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-credential-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.451528 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.451508    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dab0473-fa74-460f-8b18-69874157c8fa-istiod-ca-cert\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.453183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.453155    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-istio-envoy\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.453295 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.453236    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dab0473-fa74-460f-8b18-69874157c8fa-istio-podinfo\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.528978 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.528905    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dab0473-fa74-460f-8b18-69874157c8fa-istio-token\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.529381 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.529355    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9bjx\" (UniqueName: \"kubernetes.io/projected/4dab0473-fa74-460f-8b18-69874157c8fa-kube-api-access-z9bjx\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.557880 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.557848    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:24.683993 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.683958    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:24.987826 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.987802    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"]
Oct 30 18:51:24.989815 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:51:24.989785    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dab0473_fa74_460f_8b18_69874157c8fa.slice/crio-f62b9002716833944bccf44d27fc56de639a4b2c519710fdf154ab70aa7ac98f WatchSource:0}: Error finding container f62b9002716833944bccf44d27fc56de639a4b2c519710fdf154ab70aa7ac98f: Status 404 returned error can't find the container with id f62b9002716833944bccf44d27fc56de639a4b2c519710fdf154ab70aa7ac98f
Oct 30 18:51:24.992331 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.992299    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:51:24.992442 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.992380    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:51:24.992442 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:24.992428    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:51:25.778100 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:25.778052    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv" event={"ID":"4dab0473-fa74-460f-8b18-69874157c8fa","Type":"ContainerStarted","Data":"8d2c6380a2d90f3b4be291705554417d2ef61505b1c901eb7534f3058e715aef"}
Oct 30 18:51:25.778600 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:25.778126    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv" event={"ID":"4dab0473-fa74-460f-8b18-69874157c8fa","Type":"ContainerStarted","Data":"f62b9002716833944bccf44d27fc56de639a4b2c519710fdf154ab70aa7ac98f"}
Oct 30 18:51:25.892629 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:25.892580    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv" podStartSLOduration=1.89256771 podStartE2EDuration="1.89256771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:25.891496465 +0000 UTC m=+887.168546910" watchObservedRunningTime="2026-10-30 18:51:25.89256771 +0000 UTC m=+887.169618156"
Oct 30 18:51:26.558494 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:26.558454    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:26.559988 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:26.559966    4924 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 18:51:26.560088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:26.560013    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv" podUID="4dab0473-fa74-460f-8b18-69874157c8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 18:51:27.558634 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:27.558595    4924 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 18:51:27.559045 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:27.558652    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv" podUID="4dab0473-fa74-460f-8b18-69874157c8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 18:51:28.117089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:28.112087    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:28.558689 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:28.558602    4924 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 18:51:28.558689 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:28.558669    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv" podUID="4dab0473-fa74-460f-8b18-69874157c8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 18:51:29.526085 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:29.525439    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:29.559038 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:29.559006    4924 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 18:51:29.559536 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:29.559079    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv" podUID="4dab0473-fa74-460f-8b18-69874157c8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 18:51:30.255638 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:30.255590    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:30.558940 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:30.558853    4924 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 30 18:51:30.558940 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:30.558920    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv" podUID="4dab0473-fa74-460f-8b18-69874157c8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 30 18:51:31.563777 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:31.563743    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:31.564163 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:31.564143    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:31.564799 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:31.564775    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 18:51:31.756110 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:31.756079    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:51:31.768226 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:31.768203    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:51:31.867089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:31.864882    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:51:33.230656 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:33.230618    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:33.851660 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:33.851627    4924 generic.go:356] "Generic (PLEG): container finished" podID="d6ec64bd-1810-4f78-b451-9faae648be6e" containerID="8368c268d84e8ae8b1654db113e7462130487deb364760b85708c6aa49205751" exitCode=0
Oct 30 18:51:33.851832 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:33.851699    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w" event={"ID":"d6ec64bd-1810-4f78-b451-9faae648be6e","Type":"ContainerDied","Data":"8368c268d84e8ae8b1654db113e7462130487deb364760b85708c6aa49205751"}
Oct 30 18:51:33.852050 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:33.852029    4924 scope.go:122] "RemoveContainer" containerID="8368c268d84e8ae8b1654db113e7462130487deb364760b85708c6aa49205751"
Oct 30 18:51:33.853243 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:33.853217    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:51:33.853331 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:33.853281    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:51:33.853331 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:33.853311    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:51:34.598462 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:34.598424    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:34.870157 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:34.868959    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w" event={"ID":"d6ec64bd-1810-4f78-b451-9faae648be6e","Type":"ContainerStarted","Data":"46ba7f99d80df3f1099875ba25165476edd2516c73f2a949109eb012a3423062"}
Oct 30 18:51:35.195021 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:35.194923    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:35.384897 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:35.384864    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:35.627090 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:35.621405    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:51:35.627090 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:35.621523    4924 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:51:35.627090 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:35.621575    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w" podUID="d6ec64bd-1810-4f78-b451-9faae648be6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 18:51:35.719369 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:35.719332    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:36.626088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:36.622443    4924 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:51:36.626088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:36.622505    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w" podUID="d6ec64bd-1810-4f78-b451-9faae648be6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 18:51:37.627466 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:37.627430    4924 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:51:37.627868 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:37.627488    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w" podUID="d6ec64bd-1810-4f78-b451-9faae648be6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 18:51:38.623464 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:38.623422    4924 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:51:38.623766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:38.623740    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w" podUID="d6ec64bd-1810-4f78-b451-9faae648be6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 18:51:38.842957 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:38.842920    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:38.949873 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:38.949797    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 18:51:38.952668 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:38.952649    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 18:51:39.466013 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:39.465981    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:39.624089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:39.623463    4924 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:51:39.624089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:39.623541    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w" podUID="d6ec64bd-1810-4f78-b451-9faae648be6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 18:51:39.649124 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:39.649093    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:40.547209 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:40.547175    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:40.620374 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:40.620341    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:51:40.620649 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:40.620633    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:51:40.621417 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:40.621393    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 18:51:45.302637 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.302595    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:45.331106 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.331075    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"]
Oct 30 18:51:45.334425 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.334403    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:51:45.421787 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.421756    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-model-cache\") pod \"simulator-w2-ec8dab-kserve-b87777d68-4l6n2\" (UID: \"fdf4711c-418a-436b-848a-d46974566573\") " pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:51:45.421787 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.421793    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-tmp-dir\") pod \"simulator-w2-ec8dab-kserve-b87777d68-4l6n2\" (UID: \"fdf4711c-418a-436b-848a-d46974566573\") " pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:51:45.422038 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.421907    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvm88\" (UniqueName: \"kubernetes.io/projected/fdf4711c-418a-436b-848a-d46974566573-kube-api-access-tvm88\") pod \"simulator-w2-ec8dab-kserve-b87777d68-4l6n2\" (UID: \"fdf4711c-418a-436b-848a-d46974566573\") " pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:51:45.422038 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.421951    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf4711c-418a-436b-848a-d46974566573-tls-certs\") pod \"simulator-w2-ec8dab-kserve-b87777d68-4l6n2\" (UID: \"fdf4711c-418a-436b-848a-d46974566573\") " pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:51:45.422038 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.422006    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-home\") pod \"simulator-w2-ec8dab-kserve-b87777d68-4l6n2\" (UID: \"fdf4711c-418a-436b-848a-d46974566573\") " pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:51:45.422255 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.422130    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-dshm\") pod \"simulator-w2-ec8dab-kserve-b87777d68-4l6n2\" (UID: \"fdf4711c-418a-436b-848a-d46974566573\") " pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:51:45.430388 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.430360    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-ec8dab\"/\"openshift-service-ca.crt\""
Oct 30 18:51:45.430493 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.430449    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ec8dab\"/\"default-dockercfg-kvwpm\""
Oct 30 18:51:45.430493 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.430470    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ec8dab\"/\"simulator-w2-ec8dab-kserve-self-signed-certs\""
Oct 30 18:51:45.430578 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.430494    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-ec8dab\"/\"kube-root-ca.crt\""
Oct 30 18:51:45.433090 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.433074    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"]
Oct 30 18:51:45.523661 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.523627    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvm88\" (UniqueName: \"kubernetes.io/projected/fdf4711c-418a-436b-848a-d46974566573-kube-api-access-tvm88\") pod \"simulator-w2-ec8dab-kserve-b87777d68-4l6n2\" (UID: \"fdf4711c-418a-436b-848a-d46974566573\") " pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:51:45.523661 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.523665    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf4711c-418a-436b-848a-d46974566573-tls-certs\") pod \"simulator-w2-ec8dab-kserve-b87777d68-4l6n2\" (UID: \"fdf4711c-418a-436b-848a-d46974566573\") " pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:51:45.523891 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.523698    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-home\") pod \"simulator-w2-ec8dab-kserve-b87777d68-4l6n2\" (UID: \"fdf4711c-418a-436b-848a-d46974566573\") " pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:51:45.523891 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.523868    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-dshm\") pod \"simulator-w2-ec8dab-kserve-b87777d68-4l6n2\" (UID: \"fdf4711c-418a-436b-848a-d46974566573\") " pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:51:45.523965 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.523941    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-model-cache\") pod \"simulator-w2-ec8dab-kserve-b87777d68-4l6n2\" (UID: \"fdf4711c-418a-436b-848a-d46974566573\") " pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:51:45.524007 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.523979    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-tmp-dir\") pod \"simulator-w2-ec8dab-kserve-b87777d68-4l6n2\" (UID: \"fdf4711c-418a-436b-848a-d46974566573\") " pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:51:45.524252 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.524230    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-home\") pod \"simulator-w2-ec8dab-kserve-b87777d68-4l6n2\" (UID: \"fdf4711c-418a-436b-848a-d46974566573\") " pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:51:45.524448 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.524424    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-model-cache\") pod \"simulator-w2-ec8dab-kserve-b87777d68-4l6n2\" (UID: \"fdf4711c-418a-436b-848a-d46974566573\") " pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:51:45.524448 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.524442    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-tmp-dir\") pod \"simulator-w2-ec8dab-kserve-b87777d68-4l6n2\" (UID: \"fdf4711c-418a-436b-848a-d46974566573\") " pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:51:45.526142 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.526108    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-dshm\") pod \"simulator-w2-ec8dab-kserve-b87777d68-4l6n2\" (UID: \"fdf4711c-418a-436b-848a-d46974566573\") " pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:51:45.526262 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.526246    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf4711c-418a-436b-848a-d46974566573-tls-certs\") pod \"simulator-w2-ec8dab-kserve-b87777d68-4l6n2\" (UID: \"fdf4711c-418a-436b-848a-d46974566573\") " pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:51:45.536430 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.536404    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:45.578006 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.577944    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvm88\" (UniqueName: \"kubernetes.io/projected/fdf4711c-418a-436b-848a-d46974566573-kube-api-access-tvm88\") pod \"simulator-w2-ec8dab-kserve-b87777d68-4l6n2\" (UID: \"fdf4711c-418a-436b-848a-d46974566573\") " pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:51:45.644827 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:45.644805    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:51:45.718753 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:51:45.718722    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:51:46.010288 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:46.010266    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"]
Oct 30 18:51:46.011661 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:51:46.011634    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfdf4711c_418a_436b_848a_d46974566573.slice/crio-4a05f6deb91406b968c838237439280cb8ca45d514013b66962b88e3e15af471 WatchSource:0}: Error finding container 4a05f6deb91406b968c838237439280cb8ca45d514013b66962b88e3e15af471: Status 404 returned error can't find the container with id 4a05f6deb91406b968c838237439280cb8ca45d514013b66962b88e3e15af471
Oct 30 18:51:46.935547 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:46.935511    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2" event={"ID":"fdf4711c-418a-436b-848a-d46974566573","Type":"ContainerStarted","Data":"6a72e506dbbd782b6d51df437ebcbfd0e90ccd639153874a83f28c29827d33ad"}
Oct 30 18:51:46.935547 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:46.935550    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2" event={"ID":"fdf4711c-418a-436b-848a-d46974566573","Type":"ContainerStarted","Data":"4a05f6deb91406b968c838237439280cb8ca45d514013b66962b88e3e15af471"}
Oct 30 18:51:47.046467 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:47.046416    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2" podStartSLOduration=2.046402816 podStartE2EDuration="2.046402816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:47.044115132 +0000 UTC m=+908.321165576" watchObservedRunningTime="2026-10-30 18:51:47.046402816 +0000 UTC m=+908.323453260"
Oct 30 18:51:47.645156 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:47.645109    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:51:48.182784 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:48.182750    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:48.778611 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:48.778579    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:48.973618 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:48.972047    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:50.543649 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:50.543617    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:53.347183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:53.347143    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:54.685577 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:54.685537    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:55.335455 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:55.335422    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:55.537489 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:55.537458    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:55.645759 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:55.645730    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:51:55.658060 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:55.658035    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:51:55.726922 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:55.726886    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:51:56.006106 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:51:56.006011    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:52:00.326007 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:00.325973    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:04.598683 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.598642    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"]
Oct 30 18:52:04.601665 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.601647    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:52:04.651237 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.651212    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-3b6ad6\"/\"openshift-service-ca.crt\""
Oct 30 18:52:04.663794 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.663765    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drrgn\" (UniqueName: \"kubernetes.io/projected/1a90412d-c047-422e-b3c7-9546d286c303-kube-api-access-drrgn\") pod \"simulator-w5-3b6ad6-kserve-84797f4658-8sj7m\" (UID: \"1a90412d-c047-422e-b3c7-9546d286c303\") " pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:52:04.663936 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.663827    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-model-cache\") pod \"simulator-w5-3b6ad6-kserve-84797f4658-8sj7m\" (UID: \"1a90412d-c047-422e-b3c7-9546d286c303\") " pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:52:04.664006 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.663974    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-home\") pod \"simulator-w5-3b6ad6-kserve-84797f4658-8sj7m\" (UID: \"1a90412d-c047-422e-b3c7-9546d286c303\") " pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:52:04.664076 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.664028    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-dshm\") pod \"simulator-w5-3b6ad6-kserve-84797f4658-8sj7m\" (UID: \"1a90412d-c047-422e-b3c7-9546d286c303\") " pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:52:04.664131 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.664114    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a90412d-c047-422e-b3c7-9546d286c303-tls-certs\") pod \"simulator-w5-3b6ad6-kserve-84797f4658-8sj7m\" (UID: \"1a90412d-c047-422e-b3c7-9546d286c303\") " pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:52:04.664205 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.664179    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-tmp-dir\") pod \"simulator-w5-3b6ad6-kserve-84797f4658-8sj7m\" (UID: \"1a90412d-c047-422e-b3c7-9546d286c303\") " pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:52:04.697548 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.697519    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-3b6ad6\"/\"default-dockercfg-b8plq\""
Oct 30 18:52:04.697548 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.697528    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-3b6ad6\"/\"kube-root-ca.crt\""
Oct 30 18:52:04.697548 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.697554    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-3b6ad6\"/\"simulator-w5-3b6ad6-kserve-self-signed-certs\""
Oct 30 18:52:04.701637 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.701616    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"]
Oct 30 18:52:04.745509 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.745481    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:04.765230 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.765193    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-home\") pod \"simulator-w5-3b6ad6-kserve-84797f4658-8sj7m\" (UID: \"1a90412d-c047-422e-b3c7-9546d286c303\") " pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:52:04.765412 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.765247    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-dshm\") pod \"simulator-w5-3b6ad6-kserve-84797f4658-8sj7m\" (UID: \"1a90412d-c047-422e-b3c7-9546d286c303\") " pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:52:04.765412 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.765280    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a90412d-c047-422e-b3c7-9546d286c303-tls-certs\") pod \"simulator-w5-3b6ad6-kserve-84797f4658-8sj7m\" (UID: \"1a90412d-c047-422e-b3c7-9546d286c303\") " pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:52:04.765412 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.765330    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-tmp-dir\") pod \"simulator-w5-3b6ad6-kserve-84797f4658-8sj7m\" (UID: \"1a90412d-c047-422e-b3c7-9546d286c303\") " pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:52:04.765412 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.765375    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drrgn\" (UniqueName: \"kubernetes.io/projected/1a90412d-c047-422e-b3c7-9546d286c303-kube-api-access-drrgn\") pod \"simulator-w5-3b6ad6-kserve-84797f4658-8sj7m\" (UID: \"1a90412d-c047-422e-b3c7-9546d286c303\") " pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:52:04.765604 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.765452    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-model-cache\") pod \"simulator-w5-3b6ad6-kserve-84797f4658-8sj7m\" (UID: \"1a90412d-c047-422e-b3c7-9546d286c303\") " pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:52:04.765719 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.765693    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-home\") pod \"simulator-w5-3b6ad6-kserve-84797f4658-8sj7m\" (UID: \"1a90412d-c047-422e-b3c7-9546d286c303\") " pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:52:04.765846 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.765825    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-tmp-dir\") pod \"simulator-w5-3b6ad6-kserve-84797f4658-8sj7m\" (UID: \"1a90412d-c047-422e-b3c7-9546d286c303\") " pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:52:04.765910 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.765841    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-model-cache\") pod \"simulator-w5-3b6ad6-kserve-84797f4658-8sj7m\" (UID: \"1a90412d-c047-422e-b3c7-9546d286c303\") " pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:52:04.767579 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.767561    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-dshm\") pod \"simulator-w5-3b6ad6-kserve-84797f4658-8sj7m\" (UID: \"1a90412d-c047-422e-b3c7-9546d286c303\") " pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:52:04.767870 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.767852    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a90412d-c047-422e-b3c7-9546d286c303-tls-certs\") pod \"simulator-w5-3b6ad6-kserve-84797f4658-8sj7m\" (UID: \"1a90412d-c047-422e-b3c7-9546d286c303\") " pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:52:04.820257 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.820223    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drrgn\" (UniqueName: \"kubernetes.io/projected/1a90412d-c047-422e-b3c7-9546d286c303-kube-api-access-drrgn\") pod \"simulator-w5-3b6ad6-kserve-84797f4658-8sj7m\" (UID: \"1a90412d-c047-422e-b3c7-9546d286c303\") " pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:52:04.912950 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:04.912867    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:52:05.278225 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:05.278198    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"]
Oct 30 18:52:05.279626 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:52:05.279588    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a90412d_c047_422e_b3c7_9546d286c303.slice/crio-51267a37bad47f3b51bdf3f01cdb3c1d281b39d040665c3f60f78fb134fcb916 WatchSource:0}: Error finding container 51267a37bad47f3b51bdf3f01cdb3c1d281b39d040665c3f60f78fb134fcb916: Status 404 returned error can't find the container with id 51267a37bad47f3b51bdf3f01cdb3c1d281b39d040665c3f60f78fb134fcb916
Oct 30 18:52:05.312291 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:05.311989    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:05.719821 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:05.719786    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:06.052583 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:06.052490    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m" event={"ID":"1a90412d-c047-422e-b3c7-9546d286c303","Type":"ContainerStarted","Data":"297bf6e960483b78655aeb436dc75535c47c8bec446e852da809dda6fe8dffd0"}
Oct 30 18:52:06.052583 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:06.052545    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m" event={"ID":"1a90412d-c047-422e-b3c7-9546d286c303","Type":"ContainerStarted","Data":"51267a37bad47f3b51bdf3f01cdb3c1d281b39d040665c3f60f78fb134fcb916"}
Oct 30 18:52:06.163437 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:06.163392    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m" podStartSLOduration=2.163378186 podStartE2EDuration="2.163378186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:06.160550315 +0000 UTC m=+927.437600759" watchObservedRunningTime="2026-10-30 18:52:06.163378186 +0000 UTC m=+927.440428631"
Oct 30 18:52:06.326948 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:06.326864    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:06.509472 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:06.509437    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:06.912948 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:06.912920    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:52:10.109735 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:10.109686    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:14.874302 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:14.874271    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:14.913747 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:14.913710    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:52:14.926448 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:14.926423    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:52:15.118428 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:15.118399    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:52:15.711412 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:52:15.711381    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:52:17.821051 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:17.821017    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:19.857137 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:19.857049    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:23.100242 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:23.100210    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:24.541251 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:24.541220    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:29.305959 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:29.305919    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:33.048192 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.048151    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"]
Oct 30 18:52:33.055292 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.055266    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:52:33.108284 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.108250    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"]
Oct 30 18:52:33.150803 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.150770    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-75e91b\"/\"kube-root-ca.crt\""
Oct 30 18:52:33.150803 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.150770    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-75e91b\"/\"openshift-service-ca.crt\""
Oct 30 18:52:33.151595 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.151572    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-75e91b\"/\"default-dockercfg-qrgx8\""
Oct 30 18:52:33.151748 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.151605    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-75e91b\"/\"simulator-w4-75e91b-kserve-self-signed-certs\""
Oct 30 18:52:33.184102 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.184047    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-tmp-dir\") pod \"simulator-w4-75e91b-kserve-5f668ff968-kpwc5\" (UID: \"925a4de9-b451-4263-9a1c-eebbbae4a08c\") " pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:52:33.184248 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.184157    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/925a4de9-b451-4263-9a1c-eebbbae4a08c-tls-certs\") pod \"simulator-w4-75e91b-kserve-5f668ff968-kpwc5\" (UID: \"925a4de9-b451-4263-9a1c-eebbbae4a08c\") " pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:52:33.184326 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.184310    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-home\") pod \"simulator-w4-75e91b-kserve-5f668ff968-kpwc5\" (UID: \"925a4de9-b451-4263-9a1c-eebbbae4a08c\") " pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:52:33.184362 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.184342    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-dshm\") pod \"simulator-w4-75e91b-kserve-5f668ff968-kpwc5\" (UID: \"925a4de9-b451-4263-9a1c-eebbbae4a08c\") " pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:52:33.184396 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.184367    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-model-cache\") pod \"simulator-w4-75e91b-kserve-5f668ff968-kpwc5\" (UID: \"925a4de9-b451-4263-9a1c-eebbbae4a08c\") " pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:52:33.184484 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.184463    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mjzd\" (UniqueName: \"kubernetes.io/projected/925a4de9-b451-4263-9a1c-eebbbae4a08c-kube-api-access-8mjzd\") pod \"simulator-w4-75e91b-kserve-5f668ff968-kpwc5\" (UID: \"925a4de9-b451-4263-9a1c-eebbbae4a08c\") " pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:52:33.186207 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.186180    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:33.285535 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.285502    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-home\") pod \"simulator-w4-75e91b-kserve-5f668ff968-kpwc5\" (UID: \"925a4de9-b451-4263-9a1c-eebbbae4a08c\") " pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:52:33.285535 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.285540    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-dshm\") pod \"simulator-w4-75e91b-kserve-5f668ff968-kpwc5\" (UID: \"925a4de9-b451-4263-9a1c-eebbbae4a08c\") " pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:52:33.285800 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.285564    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-model-cache\") pod \"simulator-w4-75e91b-kserve-5f668ff968-kpwc5\" (UID: \"925a4de9-b451-4263-9a1c-eebbbae4a08c\") " pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:52:33.285800 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.285584    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mjzd\" (UniqueName: \"kubernetes.io/projected/925a4de9-b451-4263-9a1c-eebbbae4a08c-kube-api-access-8mjzd\") pod \"simulator-w4-75e91b-kserve-5f668ff968-kpwc5\" (UID: \"925a4de9-b451-4263-9a1c-eebbbae4a08c\") " pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:52:33.285800 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.285650    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-tmp-dir\") pod \"simulator-w4-75e91b-kserve-5f668ff968-kpwc5\" (UID: \"925a4de9-b451-4263-9a1c-eebbbae4a08c\") " pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:52:33.285800 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.285668    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/925a4de9-b451-4263-9a1c-eebbbae4a08c-tls-certs\") pod \"simulator-w4-75e91b-kserve-5f668ff968-kpwc5\" (UID: \"925a4de9-b451-4263-9a1c-eebbbae4a08c\") " pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:52:33.286033 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.286006    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-home\") pod \"simulator-w4-75e91b-kserve-5f668ff968-kpwc5\" (UID: \"925a4de9-b451-4263-9a1c-eebbbae4a08c\") " pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:52:33.286111 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.286031    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-model-cache\") pod \"simulator-w4-75e91b-kserve-5f668ff968-kpwc5\" (UID: \"925a4de9-b451-4263-9a1c-eebbbae4a08c\") " pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:52:33.286181 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.286101    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-tmp-dir\") pod \"simulator-w4-75e91b-kserve-5f668ff968-kpwc5\" (UID: \"925a4de9-b451-4263-9a1c-eebbbae4a08c\") " pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:52:33.287764 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.287738    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-dshm\") pod \"simulator-w4-75e91b-kserve-5f668ff968-kpwc5\" (UID: \"925a4de9-b451-4263-9a1c-eebbbae4a08c\") " pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:52:33.288035 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.288018    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/925a4de9-b451-4263-9a1c-eebbbae4a08c-tls-certs\") pod \"simulator-w4-75e91b-kserve-5f668ff968-kpwc5\" (UID: \"925a4de9-b451-4263-9a1c-eebbbae4a08c\") " pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:52:33.340090 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.339999    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mjzd\" (UniqueName: \"kubernetes.io/projected/925a4de9-b451-4263-9a1c-eebbbae4a08c-kube-api-access-8mjzd\") pod \"simulator-w4-75e91b-kserve-5f668ff968-kpwc5\" (UID: \"925a4de9-b451-4263-9a1c-eebbbae4a08c\") " pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:52:33.371484 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.371450    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:52:33.572904 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.572874    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:33.740011 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:33.739977    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"]
Oct 30 18:52:33.741625 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:52:33.741589    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod925a4de9_b451_4263_9a1c_eebbbae4a08c.slice/crio-8aabbbedd9a6301d65e391c9428b3b371dd3934768f9179ecc58901072ae602a WatchSource:0}: Error finding container 8aabbbedd9a6301d65e391c9428b3b371dd3934768f9179ecc58901072ae602a: Status 404 returned error can't find the container with id 8aabbbedd9a6301d65e391c9428b3b371dd3934768f9179ecc58901072ae602a
Oct 30 18:52:34.049645 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:34.049543    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:34.217145 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:34.217111    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5" event={"ID":"925a4de9-b451-4263-9a1c-eebbbae4a08c","Type":"ContainerStarted","Data":"eb96402c7314a0b89230f8b32897eb32a3f9e812c752e09395880f680cdb5aa3"}
Oct 30 18:52:34.217340 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:34.217158    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5" event={"ID":"925a4de9-b451-4263-9a1c-eebbbae4a08c","Type":"ContainerStarted","Data":"8aabbbedd9a6301d65e391c9428b3b371dd3934768f9179ecc58901072ae602a"}
Oct 30 18:52:34.325372 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:34.325276    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5" podStartSLOduration=1.325261614 podStartE2EDuration="1.325261614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:34.324172369 +0000 UTC m=+955.601222827" watchObservedRunningTime="2026-10-30 18:52:34.325261614 +0000 UTC m=+955.602312058"
Oct 30 18:52:34.371566 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:34.371534    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:52:34.618446 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:34.618414    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:38.809322 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:38.809262    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:42.313395 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:42.313359    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:43.372471 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:43.372432    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:52:43.385451 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:43.385417    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:52:43.604415 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:43.604380    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:44.279722 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:44.279695    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:52:45.710623 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:52:45.710592    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:52:48.511466 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:48.511431    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:53.247621 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:53.247580    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:56.304765 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:56.304731    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:52:58.251581 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:52:58.251381    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:03.006309 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:03.006271    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:07.836824 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:07.836785    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:12.222144 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.222111    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"]
Oct 30 18:53:12.226722 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.226687    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:53:12.294704 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.294673    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"]
Oct 30 18:53:12.316239 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.316210    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-model-cache\") pod \"premium-w0-f5705f-kserve-7854c57b49-xrck7\" (UID: \"97a3b018-9083-4b7c-af88-6edddc585d67\") " pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:53:12.316393 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.316356    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnbk6\" (UniqueName: \"kubernetes.io/projected/97a3b018-9083-4b7c-af88-6edddc585d67-kube-api-access-cnbk6\") pod \"premium-w0-f5705f-kserve-7854c57b49-xrck7\" (UID: \"97a3b018-9083-4b7c-af88-6edddc585d67\") " pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:53:12.316458 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.316447    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97a3b018-9083-4b7c-af88-6edddc585d67-tls-certs\") pod \"premium-w0-f5705f-kserve-7854c57b49-xrck7\" (UID: \"97a3b018-9083-4b7c-af88-6edddc585d67\") " pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:53:12.316504 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.316479    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-home\") pod \"premium-w0-f5705f-kserve-7854c57b49-xrck7\" (UID: \"97a3b018-9083-4b7c-af88-6edddc585d67\") " pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:53:12.316560 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.316522    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-tmp-dir\") pod \"premium-w0-f5705f-kserve-7854c57b49-xrck7\" (UID: \"97a3b018-9083-4b7c-af88-6edddc585d67\") " pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:53:12.316622 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.316595    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-dshm\") pod \"premium-w0-f5705f-kserve-7854c57b49-xrck7\" (UID: \"97a3b018-9083-4b7c-af88-6edddc585d67\") " pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:53:12.323398 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.323376    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f5705f\"/\"kube-root-ca.crt\""
Oct 30 18:53:12.323478 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.323450    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f5705f\"/\"default-dockercfg-d2rht\""
Oct 30 18:53:12.323847 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.323829    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f5705f\"/\"openshift-service-ca.crt\""
Oct 30 18:53:12.323918 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.323881    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f5705f\"/\"premium-w0-f5705f-kserve-self-signed-certs\""
Oct 30 18:53:12.363137 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.363101    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:12.417956 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.417920    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-tmp-dir\") pod \"premium-w0-f5705f-kserve-7854c57b49-xrck7\" (UID: \"97a3b018-9083-4b7c-af88-6edddc585d67\") " pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:53:12.418184 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.417964    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-dshm\") pod \"premium-w0-f5705f-kserve-7854c57b49-xrck7\" (UID: \"97a3b018-9083-4b7c-af88-6edddc585d67\") " pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:53:12.418184 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.417995    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-model-cache\") pod \"premium-w0-f5705f-kserve-7854c57b49-xrck7\" (UID: \"97a3b018-9083-4b7c-af88-6edddc585d67\") " pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:53:12.418184 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.418051    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnbk6\" (UniqueName: \"kubernetes.io/projected/97a3b018-9083-4b7c-af88-6edddc585d67-kube-api-access-cnbk6\") pod \"premium-w0-f5705f-kserve-7854c57b49-xrck7\" (UID: \"97a3b018-9083-4b7c-af88-6edddc585d67\") " pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:53:12.418184 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.418117    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97a3b018-9083-4b7c-af88-6edddc585d67-tls-certs\") pod \"premium-w0-f5705f-kserve-7854c57b49-xrck7\" (UID: \"97a3b018-9083-4b7c-af88-6edddc585d67\") " pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:53:12.418184 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.418147    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-home\") pod \"premium-w0-f5705f-kserve-7854c57b49-xrck7\" (UID: \"97a3b018-9083-4b7c-af88-6edddc585d67\") " pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:53:12.418481 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.418459    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-tmp-dir\") pod \"premium-w0-f5705f-kserve-7854c57b49-xrck7\" (UID: \"97a3b018-9083-4b7c-af88-6edddc585d67\") " pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:53:12.418531 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.418480    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-model-cache\") pod \"premium-w0-f5705f-kserve-7854c57b49-xrck7\" (UID: \"97a3b018-9083-4b7c-af88-6edddc585d67\") " pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:53:12.418566 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.418526    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-home\") pod \"premium-w0-f5705f-kserve-7854c57b49-xrck7\" (UID: \"97a3b018-9083-4b7c-af88-6edddc585d67\") " pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:53:12.420379 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.420354    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-dshm\") pod \"premium-w0-f5705f-kserve-7854c57b49-xrck7\" (UID: \"97a3b018-9083-4b7c-af88-6edddc585d67\") " pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:53:12.420693 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.420672    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97a3b018-9083-4b7c-af88-6edddc585d67-tls-certs\") pod \"premium-w0-f5705f-kserve-7854c57b49-xrck7\" (UID: \"97a3b018-9083-4b7c-af88-6edddc585d67\") " pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:53:12.472660 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.472597    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnbk6\" (UniqueName: \"kubernetes.io/projected/97a3b018-9083-4b7c-af88-6edddc585d67-kube-api-access-cnbk6\") pod \"premium-w0-f5705f-kserve-7854c57b49-xrck7\" (UID: \"97a3b018-9083-4b7c-af88-6edddc585d67\") " pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:53:12.540780 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.540753    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:53:12.649682 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.649648    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:12.905130 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:12.905105    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"]
Oct 30 18:53:12.906750 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:53:12.906722    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97a3b018_9083_4b7c_af88_6edddc585d67.slice/crio-a2c4f3ed50c3532bcbbd76c5d47d71543b05463eed799826c42ca38bca46b7e4 WatchSource:0}: Error finding container a2c4f3ed50c3532bcbbd76c5d47d71543b05463eed799826c42ca38bca46b7e4: Status 404 returned error can't find the container with id a2c4f3ed50c3532bcbbd76c5d47d71543b05463eed799826c42ca38bca46b7e4
Oct 30 18:53:13.448208 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:13.448173    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7" event={"ID":"97a3b018-9083-4b7c-af88-6edddc585d67","Type":"ContainerStarted","Data":"c8354c5d228abc52be0eeda3d9c4a9a6dcfa6133e3da3e26f846d59ba8c5538c"}
Oct 30 18:53:13.448562 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:13.448222    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7" event={"ID":"97a3b018-9083-4b7c-af88-6edddc585d67","Type":"ContainerStarted","Data":"a2c4f3ed50c3532bcbbd76c5d47d71543b05463eed799826c42ca38bca46b7e4"}
Oct 30 18:53:13.541190 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:13.541167    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:53:13.561936 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:13.561892    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7" podStartSLOduration=1.561877502 podStartE2EDuration="1.561877502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:13.559262192 +0000 UTC m=+994.836312637" watchObservedRunningTime="2026-10-30 18:53:13.561877502 +0000 UTC m=+994.838927949"
Oct 30 18:53:15.711219 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:53:15.711183    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:53:17.383878 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:17.383846    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b80ef4f4-dkwc6"]
Oct 30 18:53:17.386789 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:17.386769    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ef4f4-dkwc6"
Oct 30 18:53:17.439434 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:17.439407    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:53:17.470034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:17.470004    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b956f39-ff10-4578-a5a8-cd17cb0e71fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b80ef4f4-dkwc6\" (UID: \"3b956f39-ff10-4578-a5a8-cd17cb0e71fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ef4f4-dkwc6"
Oct 30 18:53:17.470212 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:17.470054    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b956f39-ff10-4578-a5a8-cd17cb0e71fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b80ef4f4-dkwc6\" (UID: \"3b956f39-ff10-4578-a5a8-cd17cb0e71fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ef4f4-dkwc6"
Oct 30 18:53:17.470305 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:17.470289    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvqwb\" (UniqueName: \"kubernetes.io/projected/3b956f39-ff10-4578-a5a8-cd17cb0e71fe-kube-api-access-zvqwb\") pod \"maas-api-revoke-sub-b80ef4f4-dkwc6\" (UID: \"3b956f39-ff10-4578-a5a8-cd17cb0e71fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ef4f4-dkwc6"
Oct 30 18:53:17.485297 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:17.485275    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:53:17.485297 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:17.485282    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:53:17.485480 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:17.485418    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 18:53:17.487764 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:17.487739    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b80ef4f4-dkwc6"]
Oct 30 18:53:17.508827 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:17.508803    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:17.571715 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:17.571683    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b956f39-ff10-4578-a5a8-cd17cb0e71fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b80ef4f4-dkwc6\" (UID: \"3b956f39-ff10-4578-a5a8-cd17cb0e71fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ef4f4-dkwc6"
Oct 30 18:53:17.571902 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:17.571735    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b956f39-ff10-4578-a5a8-cd17cb0e71fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b80ef4f4-dkwc6\" (UID: \"3b956f39-ff10-4578-a5a8-cd17cb0e71fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ef4f4-dkwc6"
Oct 30 18:53:17.571902 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:17.571858    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvqwb\" (UniqueName: \"kubernetes.io/projected/3b956f39-ff10-4578-a5a8-cd17cb0e71fe-kube-api-access-zvqwb\") pod \"maas-api-revoke-sub-b80ef4f4-dkwc6\" (UID: \"3b956f39-ff10-4578-a5a8-cd17cb0e71fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ef4f4-dkwc6"
Oct 30 18:53:17.572384 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:17.572351    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b956f39-ff10-4578-a5a8-cd17cb0e71fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b80ef4f4-dkwc6\" (UID: \"3b956f39-ff10-4578-a5a8-cd17cb0e71fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ef4f4-dkwc6"
Oct 30 18:53:17.574055 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:17.574036    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b956f39-ff10-4578-a5a8-cd17cb0e71fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b80ef4f4-dkwc6\" (UID: \"3b956f39-ff10-4578-a5a8-cd17cb0e71fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ef4f4-dkwc6"
Oct 30 18:53:17.626336 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:17.626306    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvqwb\" (UniqueName: \"kubernetes.io/projected/3b956f39-ff10-4578-a5a8-cd17cb0e71fe-kube-api-access-zvqwb\") pod \"maas-api-revoke-sub-b80ef4f4-dkwc6\" (UID: \"3b956f39-ff10-4578-a5a8-cd17cb0e71fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ef4f4-dkwc6"
Oct 30 18:53:17.699846 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:17.699785    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ef4f4-dkwc6"
Oct 30 18:53:18.074757 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:18.074726    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b80ef4f4-dkwc6"]
Oct 30 18:53:18.077389 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:53:18.077347    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b956f39_ff10_4578_a5a8_cd17cb0e71fe.slice/crio-9062e7175367040393c954c5839053a9479b7fd05ad858a8e621ffca16ed8a00 WatchSource:0}: Error finding container 9062e7175367040393c954c5839053a9479b7fd05ad858a8e621ffca16ed8a00: Status 404 returned error can't find the container with id 9062e7175367040393c954c5839053a9479b7fd05ad858a8e621ffca16ed8a00
Oct 30 18:53:18.475685 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:18.475651    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ef4f4-dkwc6" event={"ID":"3b956f39-ff10-4578-a5a8-cd17cb0e71fe","Type":"ContainerStarted","Data":"9062e7175367040393c954c5839053a9479b7fd05ad858a8e621ffca16ed8a00"}
Oct 30 18:53:22.317015 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:22.316971    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:22.541046 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:22.541011    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:53:22.555106 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:22.555073    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:53:23.509322 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:23.509294    4924 generic.go:356] "Generic (PLEG): container finished" podID="3b956f39-ff10-4578-a5a8-cd17cb0e71fe" containerID="ecc75a578750f62c0c46f2ec8ffb810417dcb781f3cf1659da5ef61fa574e513" exitCode=0
Oct 30 18:53:23.509700 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:23.509401    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ef4f4-dkwc6" event={"ID":"3b956f39-ff10-4578-a5a8-cd17cb0e71fe","Type":"ContainerDied","Data":"ecc75a578750f62c0c46f2ec8ffb810417dcb781f3cf1659da5ef61fa574e513"}
Oct 30 18:53:23.520770 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:23.520748    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:53:24.669809 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:24.669778    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ef4f4-dkwc6"
Oct 30 18:53:24.756460 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:24.756433    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b956f39-ff10-4578-a5a8-cd17cb0e71fe-kube-api-access-zvqwb\" (UniqueName: \"kubernetes.io/projected/3b956f39-ff10-4578-a5a8-cd17cb0e71fe-kube-api-access-zvqwb\") pod \"3b956f39-ff10-4578-a5a8-cd17cb0e71fe\" (UID: \"3b956f39-ff10-4578-a5a8-cd17cb0e71fe\") "
Oct 30 18:53:24.756657 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:24.756529    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b956f39-ff10-4578-a5a8-cd17cb0e71fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b956f39-ff10-4578-a5a8-cd17cb0e71fe-maas-api-xks-ca\") pod \"3b956f39-ff10-4578-a5a8-cd17cb0e71fe\" (UID: \"3b956f39-ff10-4578-a5a8-cd17cb0e71fe\") "
Oct 30 18:53:24.756657 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:24.756604    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b956f39-ff10-4578-a5a8-cd17cb0e71fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b956f39-ff10-4578-a5a8-cd17cb0e71fe-maas-api-openshift-ca\") pod \"3b956f39-ff10-4578-a5a8-cd17cb0e71fe\" (UID: \"3b956f39-ff10-4578-a5a8-cd17cb0e71fe\") "
Oct 30 18:53:24.756962 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:24.756935    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b956f39-ff10-4578-a5a8-cd17cb0e71fe-maas-api-openshift-ca" pod "3b956f39-ff10-4578-a5a8-cd17cb0e71fe" (UID: "3b956f39-ff10-4578-a5a8-cd17cb0e71fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:24.758891 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:24.758857    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b956f39-ff10-4578-a5a8-cd17cb0e71fe-maas-api-xks-ca" pod "3b956f39-ff10-4578-a5a8-cd17cb0e71fe" (UID: "3b956f39-ff10-4578-a5a8-cd17cb0e71fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:53:24.759097 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:24.759055    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b956f39-ff10-4578-a5a8-cd17cb0e71fe-kube-api-access-zvqwb" pod "3b956f39-ff10-4578-a5a8-cd17cb0e71fe" (UID: "3b956f39-ff10-4578-a5a8-cd17cb0e71fe"). InnerVolumeSpecName "kube-api-access-zvqwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:24.770605 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:24.764679    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:24.857938 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:24.857897    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b956f39-ff10-4578-a5a8-cd17cb0e71fe-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:24.857938 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:24.857940    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvqwb\" (UniqueName: \"kubernetes.io/projected/3b956f39-ff10-4578-a5a8-cd17cb0e71fe-kube-api-access-zvqwb\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:24.858152 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:24.857956    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b956f39-ff10-4578-a5a8-cd17cb0e71fe-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:25.521773 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:25.521746    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ef4f4-dkwc6"
Oct 30 18:53:25.521945 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:25.521751    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b80ef4f4-dkwc6" event={"ID":"3b956f39-ff10-4578-a5a8-cd17cb0e71fe","Type":"ContainerDied","Data":"9062e7175367040393c954c5839053a9479b7fd05ad858a8e621ffca16ed8a00"}
Oct 30 18:53:25.521945 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:25.521857    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9062e7175367040393c954c5839053a9479b7fd05ad858a8e621ffca16ed8a00"
Oct 30 18:53:27.127504 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:27.127473    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:32.028175 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:32.028131    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:33.911714 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:33.911675    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:36.848832 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:36.846166    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:38.912849 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:38.912809    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"]
Oct 30 18:53:38.913471 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:38.913450    4924 cpu_manager.go:414] "containerMap: removing container" podUID="3b956f39-ff10-4578-a5a8-cd17cb0e71fe" containerName="revoke-keys"
Oct 30 18:53:38.913560 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:38.913472    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b956f39-ff10-4578-a5a8-cd17cb0e71fe" containerName="revoke-keys"
Oct 30 18:53:38.913623 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:38.913604    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b956f39-ff10-4578-a5a8-cd17cb0e71fe" containerName="revoke-keys"
Oct 30 18:53:38.916283 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:38.916266    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:38.966079 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:38.966031    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5f80ac-openshift-default-dockercfg-5qbv4\""
Oct 30 18:53:39.009623 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.009594    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-data\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.009776 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.009651    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5xvw\" (UniqueName: \"kubernetes.io/projected/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-kube-api-access-n5xvw\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.009776 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.009739    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-token\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.009860 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.009788    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-workload-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.009860 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.009821    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-podinfo\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.010019 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.009991    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-envoy\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.010123 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.010098    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istiod-ca-cert\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.010168 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.010155    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-credential-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.010224 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.010197    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-workload-certs\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.014306 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.014286    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"]
Oct 30 18:53:39.110771 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.110740    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-workload-certs\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.110923 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.110815    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-data\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.110923 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.110872    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5xvw\" (UniqueName: \"kubernetes.io/projected/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-kube-api-access-n5xvw\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.110923 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.110907    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-token\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.111189 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.111155    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-workload-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.111329 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.111190    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-workload-certs\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.111329 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.111203    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-data\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.111329 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.111228    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-podinfo\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.111329 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.111275    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-envoy\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.111557 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.111334    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istiod-ca-cert\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.111557 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.111379    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-credential-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.111557 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.111439    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-workload-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.111738 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.111720    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-credential-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.112117 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.112094    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istiod-ca-cert\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.113597 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.113574    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-envoy\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.113694 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.113679    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-podinfo\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.165396 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.165338    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5xvw\" (UniqueName: \"kubernetes.io/projected/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-kube-api-access-n5xvw\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.165396 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.165340    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-token\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.228974 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.228948    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:39.654574 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.654550    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"]
Oct 30 18:53:39.655877 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:53:39.655847    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81ff6d8_e46d_467e_b3fe_49d3e95b34f2.slice/crio-094d5a80a63a9e7c26b9dc80cab538c6e69baeff5e876b722e263bbb97cf1df1 WatchSource:0}: Error finding container 094d5a80a63a9e7c26b9dc80cab538c6e69baeff5e876b722e263bbb97cf1df1: Status 404 returned error can't find the container with id 094d5a80a63a9e7c26b9dc80cab538c6e69baeff5e876b722e263bbb97cf1df1
Oct 30 18:53:39.658409 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.658379    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:53:39.658487 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.658436    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:53:39.658487 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:39.658465    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:53:40.632396 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:40.632358    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4" event={"ID":"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2","Type":"ContainerStarted","Data":"1912e82c51a53e10f18c7eb541e95ea34ed4712cdad45a9bee99d24ace3a00de"}
Oct 30 18:53:40.632768 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:40.632420    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4" event={"ID":"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2","Type":"ContainerStarted","Data":"094d5a80a63a9e7c26b9dc80cab538c6e69baeff5e876b722e263bbb97cf1df1"}
Oct 30 18:53:40.841561 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:40.841514    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4" podStartSLOduration=2.841502156 podStartE2EDuration="2.841502156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:40.794658726 +0000 UTC m=+1022.071709173" watchObservedRunningTime="2026-10-30 18:53:40.841502156 +0000 UTC m=+1022.118552600"
Oct 30 18:53:41.229763 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:41.229725    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:41.231388 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:41.231367    4924 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 18:53:41.231440 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:41.231416    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4" podUID="c81ff6d8-e46d-467e-b3fe-49d3e95b34f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 18:53:41.833837 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:41.833789    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:42.229593 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:42.229554    4924 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 18:53:42.229776 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:42.229621    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4" podUID="c81ff6d8-e46d-467e-b3fe-49d3e95b34f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 18:53:42.829660 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:42.829622    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:43.230324 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:43.230285    4924 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 18:53:43.230913 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:43.230347    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4" podUID="c81ff6d8-e46d-467e-b3fe-49d3e95b34f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 18:53:44.229309 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:44.229274    4924 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 18:53:44.229513 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:44.229333    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4" podUID="c81ff6d8-e46d-467e-b3fe-49d3e95b34f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 18:53:45.229682 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:45.229647    4924 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 18:53:45.230189 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:45.229708    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4" podUID="c81ff6d8-e46d-467e-b3fe-49d3e95b34f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 18:53:45.711694 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:53:45.711658    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:53:46.407339 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:46.407304    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:46.408218 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:46.408184    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:46.409230 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:46.409211    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 18:53:46.990951 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:46.990904    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:47.401380 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:47.401340    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4c5f6b6-pc8ww"]
Oct 30 18:53:47.406578 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:47.406556    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c5f6b6-pc8ww"
Oct 30 18:53:47.457136 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:47.457103    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:53:47.457653 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:47.457263    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:53:47.457653 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:47.457352    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 18:53:47.457653 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:47.457399    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:53:47.459973 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:47.459942    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4c5f6b6-pc8ww"]
Oct 30 18:53:47.500013 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:47.499979    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb247e2-a8a0-49b9-9f12-bdfd0f494b68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4c5f6b6-pc8ww\" (UID: \"dbb247e2-a8a0-49b9-9f12-bdfd0f494b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c5f6b6-pc8ww"
Oct 30 18:53:47.500226 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:47.500052    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb247e2-a8a0-49b9-9f12-bdfd0f494b68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4c5f6b6-pc8ww\" (UID: \"dbb247e2-a8a0-49b9-9f12-bdfd0f494b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c5f6b6-pc8ww"
Oct 30 18:53:47.500226 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:47.500170    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2wj5\" (UniqueName: \"kubernetes.io/projected/dbb247e2-a8a0-49b9-9f12-bdfd0f494b68-kube-api-access-c2wj5\") pod \"maas-api-revoke-sub-b4c5f6b6-pc8ww\" (UID: \"dbb247e2-a8a0-49b9-9f12-bdfd0f494b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c5f6b6-pc8ww"
Oct 30 18:53:47.601803 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:47.601767    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb247e2-a8a0-49b9-9f12-bdfd0f494b68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4c5f6b6-pc8ww\" (UID: \"dbb247e2-a8a0-49b9-9f12-bdfd0f494b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c5f6b6-pc8ww"
Oct 30 18:53:47.601985 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:47.601870    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb247e2-a8a0-49b9-9f12-bdfd0f494b68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4c5f6b6-pc8ww\" (UID: \"dbb247e2-a8a0-49b9-9f12-bdfd0f494b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c5f6b6-pc8ww"
Oct 30 18:53:47.601985 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:47.601920    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2wj5\" (UniqueName: \"kubernetes.io/projected/dbb247e2-a8a0-49b9-9f12-bdfd0f494b68-kube-api-access-c2wj5\") pod \"maas-api-revoke-sub-b4c5f6b6-pc8ww\" (UID: \"dbb247e2-a8a0-49b9-9f12-bdfd0f494b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c5f6b6-pc8ww"
Oct 30 18:53:47.602586 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:47.602566    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb247e2-a8a0-49b9-9f12-bdfd0f494b68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4c5f6b6-pc8ww\" (UID: \"dbb247e2-a8a0-49b9-9f12-bdfd0f494b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c5f6b6-pc8ww"
Oct 30 18:53:47.604275 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:47.604255    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb247e2-a8a0-49b9-9f12-bdfd0f494b68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4c5f6b6-pc8ww\" (UID: \"dbb247e2-a8a0-49b9-9f12-bdfd0f494b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c5f6b6-pc8ww"
Oct 30 18:53:47.658040 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:47.657975    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2wj5\" (UniqueName: \"kubernetes.io/projected/dbb247e2-a8a0-49b9-9f12-bdfd0f494b68-kube-api-access-c2wj5\") pod \"maas-api-revoke-sub-b4c5f6b6-pc8ww\" (UID: \"dbb247e2-a8a0-49b9-9f12-bdfd0f494b68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c5f6b6-pc8ww"
Oct 30 18:53:47.722229 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:47.722200    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c5f6b6-pc8ww"
Oct 30 18:53:48.088567 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:48.088545    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4c5f6b6-pc8ww"]
Oct 30 18:53:48.090896 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:53:48.090864    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb247e2_a8a0_49b9_9f12_bdfd0f494b68.slice/crio-57ad00433f0719426741c9850556a9b788076a3b7de42ac3a5be59901974bdae WatchSource:0}: Error finding container 57ad00433f0719426741c9850556a9b788076a3b7de42ac3a5be59901974bdae: Status 404 returned error can't find the container with id 57ad00433f0719426741c9850556a9b788076a3b7de42ac3a5be59901974bdae
Oct 30 18:53:48.679500 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:48.679413    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c5f6b6-pc8ww" event={"ID":"dbb247e2-a8a0-49b9-9f12-bdfd0f494b68","Type":"ContainerStarted","Data":"3853bc6b174fe27b5ff888b539bdbf7d3537e8b09867ea2d1b338762bd367cc1"}
Oct 30 18:53:48.679500 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:48.679472    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c5f6b6-pc8ww" event={"ID":"dbb247e2-a8a0-49b9-9f12-bdfd0f494b68","Type":"ContainerStarted","Data":"57ad00433f0719426741c9850556a9b788076a3b7de42ac3a5be59901974bdae"}
Oct 30 18:53:48.790918 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:48.790862    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c5f6b6-pc8ww" podStartSLOduration=1.790848115 podStartE2EDuration="1.790848115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:48.788668587 +0000 UTC m=+1030.065719031" watchObservedRunningTime="2026-10-30 18:53:48.790848115 +0000 UTC m=+1030.067898560"
Oct 30 18:53:49.686831 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:49.686795    4924 generic.go:356] "Generic (PLEG): container finished" podID="dbb247e2-a8a0-49b9-9f12-bdfd0f494b68" containerID="3853bc6b174fe27b5ff888b539bdbf7d3537e8b09867ea2d1b338762bd367cc1" exitCode=0
Oct 30 18:53:49.687233 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:49.686885    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c5f6b6-pc8ww" event={"ID":"dbb247e2-a8a0-49b9-9f12-bdfd0f494b68","Type":"ContainerDied","Data":"3853bc6b174fe27b5ff888b539bdbf7d3537e8b09867ea2d1b338762bd367cc1"}
Oct 30 18:53:50.850052 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:50.850029    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c5f6b6-pc8ww"
Oct 30 18:53:50.938440 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:50.938409    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbb247e2-a8a0-49b9-9f12-bdfd0f494b68-kube-api-access-c2wj5\" (UniqueName: \"kubernetes.io/projected/dbb247e2-a8a0-49b9-9f12-bdfd0f494b68-kube-api-access-c2wj5\") pod \"dbb247e2-a8a0-49b9-9f12-bdfd0f494b68\" (UID: \"dbb247e2-a8a0-49b9-9f12-bdfd0f494b68\") "
Oct 30 18:53:50.938597 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:50.938556    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbb247e2-a8a0-49b9-9f12-bdfd0f494b68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb247e2-a8a0-49b9-9f12-bdfd0f494b68-maas-api-xks-ca\") pod \"dbb247e2-a8a0-49b9-9f12-bdfd0f494b68\" (UID: \"dbb247e2-a8a0-49b9-9f12-bdfd0f494b68\") "
Oct 30 18:53:50.938643 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:50.938608    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbb247e2-a8a0-49b9-9f12-bdfd0f494b68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb247e2-a8a0-49b9-9f12-bdfd0f494b68-maas-api-openshift-ca\") pod \"dbb247e2-a8a0-49b9-9f12-bdfd0f494b68\" (UID: \"dbb247e2-a8a0-49b9-9f12-bdfd0f494b68\") "
Oct 30 18:53:50.939127 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:50.938975    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb247e2-a8a0-49b9-9f12-bdfd0f494b68-maas-api-openshift-ca" pod "dbb247e2-a8a0-49b9-9f12-bdfd0f494b68" (UID: "dbb247e2-a8a0-49b9-9f12-bdfd0f494b68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:50.940562 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:50.940535    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb247e2-a8a0-49b9-9f12-bdfd0f494b68-kube-api-access-c2wj5" pod "dbb247e2-a8a0-49b9-9f12-bdfd0f494b68" (UID: "dbb247e2-a8a0-49b9-9f12-bdfd0f494b68"). InnerVolumeSpecName "kube-api-access-c2wj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:50.940640 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:50.940611    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbb247e2-a8a0-49b9-9f12-bdfd0f494b68-maas-api-xks-ca" pod "dbb247e2-a8a0-49b9-9f12-bdfd0f494b68" (UID: "dbb247e2-a8a0-49b9-9f12-bdfd0f494b68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:53:51.039921 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:51.039891    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2wj5\" (UniqueName: \"kubernetes.io/projected/dbb247e2-a8a0-49b9-9f12-bdfd0f494b68-kube-api-access-c2wj5\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:51.039921 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:51.039918    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb247e2-a8a0-49b9-9f12-bdfd0f494b68-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:51.039921 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:51.039928    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb247e2-a8a0-49b9-9f12-bdfd0f494b68-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:51.701118 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:51.701082    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c5f6b6-pc8ww" event={"ID":"dbb247e2-a8a0-49b9-9f12-bdfd0f494b68","Type":"ContainerDied","Data":"57ad00433f0719426741c9850556a9b788076a3b7de42ac3a5be59901974bdae"}
Oct 30 18:53:51.701118 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:51.701120    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4c5f6b6-pc8ww"
Oct 30 18:53:51.701325 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:51.701134    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57ad00433f0719426741c9850556a9b788076a3b7de42ac3a5be59901974bdae"
Oct 30 18:53:51.867310 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:51.867275    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:54.137754 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.137724    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74304d45-69fcr"]
Oct 30 18:53:54.138275 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.138262    4924 cpu_manager.go:414] "containerMap: removing container" podUID="dbb247e2-a8a0-49b9-9f12-bdfd0f494b68" containerName="revoke-keys"
Oct 30 18:53:54.138275 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.138276    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbb247e2-a8a0-49b9-9f12-bdfd0f494b68" containerName="revoke-keys"
Oct 30 18:53:54.138367 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.138358    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbb247e2-a8a0-49b9-9f12-bdfd0f494b68" containerName="revoke-keys"
Oct 30 18:53:54.146335 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.146315    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74304d45-69fcr"
Oct 30 18:53:54.195991 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.195944    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:53:54.196670 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.196643    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74304d45-69fcr"]
Oct 30 18:53:54.197528 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.197509    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:53:54.197641 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.197569    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 18:53:54.197641 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.197586    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:53:54.277410 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.277383    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c292\" (UniqueName: \"kubernetes.io/projected/0f645db0-9c78-4e62-b10e-1724821f9315-kube-api-access-5c292\") pod \"maas-api-revoke-sub-74304d45-69fcr\" (UID: \"0f645db0-9c78-4e62-b10e-1724821f9315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74304d45-69fcr"
Oct 30 18:53:54.277555 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.277458    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f645db0-9c78-4e62-b10e-1724821f9315-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74304d45-69fcr\" (UID: \"0f645db0-9c78-4e62-b10e-1724821f9315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74304d45-69fcr"
Oct 30 18:53:54.277599 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.277575    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f645db0-9c78-4e62-b10e-1724821f9315-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74304d45-69fcr\" (UID: \"0f645db0-9c78-4e62-b10e-1724821f9315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74304d45-69fcr"
Oct 30 18:53:54.378687 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.378660    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f645db0-9c78-4e62-b10e-1724821f9315-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74304d45-69fcr\" (UID: \"0f645db0-9c78-4e62-b10e-1724821f9315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74304d45-69fcr"
Oct 30 18:53:54.378829 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.378728    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f645db0-9c78-4e62-b10e-1724821f9315-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74304d45-69fcr\" (UID: \"0f645db0-9c78-4e62-b10e-1724821f9315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74304d45-69fcr"
Oct 30 18:53:54.378883 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.378824    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c292\" (UniqueName: \"kubernetes.io/projected/0f645db0-9c78-4e62-b10e-1724821f9315-kube-api-access-5c292\") pod \"maas-api-revoke-sub-74304d45-69fcr\" (UID: \"0f645db0-9c78-4e62-b10e-1724821f9315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74304d45-69fcr"
Oct 30 18:53:54.379402 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.379376    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f645db0-9c78-4e62-b10e-1724821f9315-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74304d45-69fcr\" (UID: \"0f645db0-9c78-4e62-b10e-1724821f9315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74304d45-69fcr"
Oct 30 18:53:54.381121 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.381097    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f645db0-9c78-4e62-b10e-1724821f9315-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74304d45-69fcr\" (UID: \"0f645db0-9c78-4e62-b10e-1724821f9315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74304d45-69fcr"
Oct 30 18:53:54.433275 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.433222    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c292\" (UniqueName: \"kubernetes.io/projected/0f645db0-9c78-4e62-b10e-1724821f9315-kube-api-access-5c292\") pod \"maas-api-revoke-sub-74304d45-69fcr\" (UID: \"0f645db0-9c78-4e62-b10e-1724821f9315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74304d45-69fcr"
Oct 30 18:53:54.459947 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.459923    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74304d45-69fcr"
Oct 30 18:53:54.730292 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.730217    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"]
Oct 30 18:53:54.736457 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.736432    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.781947 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.781913    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-workload-certs\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.782123 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.781993    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-workload-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.782123 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.782019    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-envoy\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.782242 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.782146    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-data\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.782242 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.782219    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-credential-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.782329 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.782260    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eaee46c-23ef-410a-a5b6-450eba9711cc-istiod-ca-cert\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.782390 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.782358    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-token\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.782471 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.782457    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs7z9\" (UniqueName: \"kubernetes.io/projected/2eaee46c-23ef-410a-a5b6-450eba9711cc-kube-api-access-zs7z9\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.782526 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.782511    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-podinfo\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.788670 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.788650    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-ec8dab-openshift-default-dockercfg-g5szs\""
Oct 30 18:53:54.792458 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.792430    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"]
Oct 30 18:53:54.852126 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.839200    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74304d45-69fcr"]
Oct 30 18:53:54.883396 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.883370    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-data\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.883497 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.883417    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-credential-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.883497 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.883443    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eaee46c-23ef-410a-a5b6-450eba9711cc-istiod-ca-cert\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.883497 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.883489    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-token\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.883660 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.883566    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs7z9\" (UniqueName: \"kubernetes.io/projected/2eaee46c-23ef-410a-a5b6-450eba9711cc-kube-api-access-zs7z9\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.883660 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.883617    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-podinfo\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.883771 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.883714    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-workload-certs\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.883771 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.883737    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-data\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.883878 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.883813    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-workload-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.883878 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.883825    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-credential-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.883983 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.883875    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-envoy\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.884109 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.884055    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-workload-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.884220 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.884198    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eaee46c-23ef-410a-a5b6-450eba9711cc-istiod-ca-cert\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.884590 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.884564    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-workload-certs\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.886352 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.886326    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-podinfo\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.886352 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.886348    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-envoy\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.937954 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.937923    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-token\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:54.938196 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:54.938179    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs7z9\" (UniqueName: \"kubernetes.io/projected/2eaee46c-23ef-410a-a5b6-450eba9711cc-kube-api-access-zs7z9\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:55.055922 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:55.055900    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:55.468816 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:55.468789    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"]
Oct 30 18:53:55.470958 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:53:55.470927    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eaee46c_23ef_410a_a5b6_450eba9711cc.slice/crio-2269e6f93fddeff5fd8aca8fc707cbf6ba3a758789678f52ec66286bde0f0786 WatchSource:0}: Error finding container 2269e6f93fddeff5fd8aca8fc707cbf6ba3a758789678f52ec66286bde0f0786: Status 404 returned error can't find the container with id 2269e6f93fddeff5fd8aca8fc707cbf6ba3a758789678f52ec66286bde0f0786
Oct 30 18:53:55.473982 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:55.473948    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:53:55.474104 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:55.474030    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:53:55.474172 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:55.474113    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:53:55.729110 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:55.728779    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p" event={"ID":"2eaee46c-23ef-410a-a5b6-450eba9711cc","Type":"ContainerStarted","Data":"b82e787d0d3b24eff24c9b84268fd81e2d3db47b25dbffb04823654bdf6863ca"}
Oct 30 18:53:55.729110 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:55.728843    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p" event={"ID":"2eaee46c-23ef-410a-a5b6-450eba9711cc","Type":"ContainerStarted","Data":"2269e6f93fddeff5fd8aca8fc707cbf6ba3a758789678f52ec66286bde0f0786"}
Oct 30 18:53:55.730845 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:55.730805    4924 generic.go:356] "Generic (PLEG): container finished" podID="0f645db0-9c78-4e62-b10e-1724821f9315" containerID="1e3d04566a88984835bb08093310ac996d1415a0262ea8cd3fa33739774252f7" exitCode=0
Oct 30 18:53:55.730960 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:55.730922    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74304d45-69fcr" event={"ID":"0f645db0-9c78-4e62-b10e-1724821f9315","Type":"ContainerDied","Data":"1e3d04566a88984835bb08093310ac996d1415a0262ea8cd3fa33739774252f7"}
Oct 30 18:53:55.731011 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:55.730963    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74304d45-69fcr" event={"ID":"0f645db0-9c78-4e62-b10e-1724821f9315","Type":"ContainerStarted","Data":"44a549fb18ebf4ce76b9ced8e45fe8a98018e98097df449b83407de391c00776"}
Oct 30 18:53:55.844013 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:55.843950    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p" podStartSLOduration=1.843920224 podStartE2EDuration="1.843920224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:55.840867301 +0000 UTC m=+1037.117917747" watchObservedRunningTime="2026-10-30 18:53:55.843920224 +0000 UTC m=+1037.120970671"
Oct 30 18:53:56.059348 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:56.059223    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:53:56.061859 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:56.061739    4924 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 18:53:56.062128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:56.062101    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p" podUID="2eaee46c-23ef-410a-a5b6-450eba9711cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 18:53:56.625084 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:56.624714    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:53:57.051092 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:57.050693    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74304d45-69fcr"
Oct 30 18:53:57.056840 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:57.056545    4924 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 18:53:57.056840 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:57.056591    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p" podUID="2eaee46c-23ef-410a-a5b6-450eba9711cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 18:53:57.166774 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:57.166741    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f645db0-9c78-4e62-b10e-1724821f9315-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f645db0-9c78-4e62-b10e-1724821f9315-maas-api-openshift-ca\") pod \"0f645db0-9c78-4e62-b10e-1724821f9315\" (UID: \"0f645db0-9c78-4e62-b10e-1724821f9315\") "
Oct 30 18:53:57.167159 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:57.167121    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f645db0-9c78-4e62-b10e-1724821f9315-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f645db0-9c78-4e62-b10e-1724821f9315-maas-api-xks-ca\") pod \"0f645db0-9c78-4e62-b10e-1724821f9315\" (UID: \"0f645db0-9c78-4e62-b10e-1724821f9315\") "
Oct 30 18:53:57.167382 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:57.167364    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f645db0-9c78-4e62-b10e-1724821f9315-kube-api-access-5c292\" (UniqueName: \"kubernetes.io/projected/0f645db0-9c78-4e62-b10e-1724821f9315-kube-api-access-5c292\") pod \"0f645db0-9c78-4e62-b10e-1724821f9315\" (UID: \"0f645db0-9c78-4e62-b10e-1724821f9315\") "
Oct 30 18:53:57.170414 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:57.170385    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f645db0-9c78-4e62-b10e-1724821f9315-maas-api-openshift-ca" pod "0f645db0-9c78-4e62-b10e-1724821f9315" (UID: "0f645db0-9c78-4e62-b10e-1724821f9315"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:53:57.187801 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:57.180597    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f645db0-9c78-4e62-b10e-1724821f9315-maas-api-xks-ca" pod "0f645db0-9c78-4e62-b10e-1724821f9315" (UID: "0f645db0-9c78-4e62-b10e-1724821f9315"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:53:57.189278 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:57.189249    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f645db0-9c78-4e62-b10e-1724821f9315-kube-api-access-5c292" pod "0f645db0-9c78-4e62-b10e-1724821f9315" (UID: "0f645db0-9c78-4e62-b10e-1724821f9315"). InnerVolumeSpecName "kube-api-access-5c292". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:57.271145 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:57.270923    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f645db0-9c78-4e62-b10e-1724821f9315-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:57.271145 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:57.270963    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f645db0-9c78-4e62-b10e-1724821f9315-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:57.271145 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:57.270976    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c292\" (UniqueName: \"kubernetes.io/projected/0f645db0-9c78-4e62-b10e-1724821f9315-kube-api-access-5c292\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:57.753243 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:57.753213    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74304d45-69fcr"
Oct 30 18:53:57.753717 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:57.753253    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74304d45-69fcr" event={"ID":"0f645db0-9c78-4e62-b10e-1724821f9315","Type":"ContainerDied","Data":"44a549fb18ebf4ce76b9ced8e45fe8a98018e98097df449b83407de391c00776"}
Oct 30 18:53:57.753717 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:57.753310    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44a549fb18ebf4ce76b9ced8e45fe8a98018e98097df449b83407de391c00776"
Oct 30 18:53:58.056787 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:58.056691    4924 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 18:53:58.056787 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:58.056760    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p" podUID="2eaee46c-23ef-410a-a5b6-450eba9711cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 18:53:59.057218 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:59.057184    4924 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 18:53:59.057688 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:53:59.057241    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p" podUID="2eaee46c-23ef-410a-a5b6-450eba9711cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 18:54:00.057172 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:00.057135    4924 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 18:54:00.057326 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:00.057188    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p" podUID="2eaee46c-23ef-410a-a5b6-450eba9711cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 18:54:01.056875 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:01.056841    4924 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 18:54:01.057047 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:01.056892    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p" podUID="2eaee46c-23ef-410a-a5b6-450eba9711cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 18:54:01.349164 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:01.349110    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:01.953883 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:01.953848    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:02.330129 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:02.330022    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:54:02.330540 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:02.330517    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:54:02.331387 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:02.331366    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 18:54:02.559714 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:02.559679    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcb25626-pr6sm"]
Oct 30 18:54:02.560216 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:02.560201    4924 cpu_manager.go:414] "containerMap: removing container" podUID="0f645db0-9c78-4e62-b10e-1724821f9315" containerName="revoke-keys"
Oct 30 18:54:02.560265 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:02.560217    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f645db0-9c78-4e62-b10e-1724821f9315" containerName="revoke-keys"
Oct 30 18:54:02.560317 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:02.560308    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f645db0-9c78-4e62-b10e-1724821f9315" containerName="revoke-keys"
Oct 30 18:54:02.563617 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:02.563598    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb25626-pr6sm"
Oct 30 18:54:02.624117 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:02.618043    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcb25626-pr6sm"]
Oct 30 18:54:02.626436 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:02.626407    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:54:02.626592 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:02.626528    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:54:02.626678 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:02.626668    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 18:54:02.626736 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:02.626717    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:54:02.655200 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:02.655143    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8b2909a-6460-47d2-99f2-7fe93b21e81c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcb25626-pr6sm\" (UID: \"e8b2909a-6460-47d2-99f2-7fe93b21e81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb25626-pr6sm"
Oct 30 18:54:02.655371 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:02.655337    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6vmx\" (UniqueName: \"kubernetes.io/projected/e8b2909a-6460-47d2-99f2-7fe93b21e81c-kube-api-access-d6vmx\") pod \"maas-api-revoke-sub-bcb25626-pr6sm\" (UID: \"e8b2909a-6460-47d2-99f2-7fe93b21e81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb25626-pr6sm"
Oct 30 18:54:02.655499 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:02.655474    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8b2909a-6460-47d2-99f2-7fe93b21e81c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcb25626-pr6sm\" (UID: \"e8b2909a-6460-47d2-99f2-7fe93b21e81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb25626-pr6sm"
Oct 30 18:54:02.756908 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:02.756872    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6vmx\" (UniqueName: \"kubernetes.io/projected/e8b2909a-6460-47d2-99f2-7fe93b21e81c-kube-api-access-d6vmx\") pod \"maas-api-revoke-sub-bcb25626-pr6sm\" (UID: \"e8b2909a-6460-47d2-99f2-7fe93b21e81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb25626-pr6sm"
Oct 30 18:54:02.757127 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:02.756949    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8b2909a-6460-47d2-99f2-7fe93b21e81c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcb25626-pr6sm\" (UID: \"e8b2909a-6460-47d2-99f2-7fe93b21e81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb25626-pr6sm"
Oct 30 18:54:02.757127 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:02.757007    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8b2909a-6460-47d2-99f2-7fe93b21e81c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcb25626-pr6sm\" (UID: \"e8b2909a-6460-47d2-99f2-7fe93b21e81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb25626-pr6sm"
Oct 30 18:54:02.757624 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:02.757601    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8b2909a-6460-47d2-99f2-7fe93b21e81c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcb25626-pr6sm\" (UID: \"e8b2909a-6460-47d2-99f2-7fe93b21e81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb25626-pr6sm"
Oct 30 18:54:02.759346 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:02.759325    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8b2909a-6460-47d2-99f2-7fe93b21e81c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcb25626-pr6sm\" (UID: \"e8b2909a-6460-47d2-99f2-7fe93b21e81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb25626-pr6sm"
Oct 30 18:54:02.812441 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:02.812408    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6vmx\" (UniqueName: \"kubernetes.io/projected/e8b2909a-6460-47d2-99f2-7fe93b21e81c-kube-api-access-d6vmx\") pod \"maas-api-revoke-sub-bcb25626-pr6sm\" (UID: \"e8b2909a-6460-47d2-99f2-7fe93b21e81c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb25626-pr6sm"
Oct 30 18:54:02.876523 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:02.876446    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb25626-pr6sm"
Oct 30 18:54:03.242030 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:03.241995    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcb25626-pr6sm"]
Oct 30 18:54:03.244110 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:54:03.244082    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b2909a_6460_47d2_99f2_7fe93b21e81c.slice/crio-a8a09fed837e2a1ba63145527fcebe010d376f269e7d13d08f23f1b471171ea3 WatchSource:0}: Error finding container a8a09fed837e2a1ba63145527fcebe010d376f269e7d13d08f23f1b471171ea3: Status 404 returned error can't find the container with id a8a09fed837e2a1ba63145527fcebe010d376f269e7d13d08f23f1b471171ea3
Oct 30 18:54:03.791708 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:03.791628    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb25626-pr6sm" event={"ID":"e8b2909a-6460-47d2-99f2-7fe93b21e81c","Type":"ContainerStarted","Data":"414a5ced4b723aa7e8a5a55881774d1fd6e8a86e702399e7008b5281e1384ecd"}
Oct 30 18:54:03.791708 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:03.791684    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb25626-pr6sm" event={"ID":"e8b2909a-6460-47d2-99f2-7fe93b21e81c","Type":"ContainerStarted","Data":"a8a09fed837e2a1ba63145527fcebe010d376f269e7d13d08f23f1b471171ea3"}
Oct 30 18:54:03.899478 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:03.899429    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb25626-pr6sm" podStartSLOduration=1.899414873 podStartE2EDuration="1.899414873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:03.898513248 +0000 UTC m=+1045.175563693" watchObservedRunningTime="2026-10-30 18:54:03.899414873 +0000 UTC m=+1045.176465314"
Oct 30 18:54:04.801055 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:04.801020    4924 generic.go:356] "Generic (PLEG): container finished" podID="e8b2909a-6460-47d2-99f2-7fe93b21e81c" containerID="414a5ced4b723aa7e8a5a55881774d1fd6e8a86e702399e7008b5281e1384ecd" exitCode=0
Oct 30 18:54:04.801527 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:04.801104    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb25626-pr6sm" event={"ID":"e8b2909a-6460-47d2-99f2-7fe93b21e81c","Type":"ContainerDied","Data":"414a5ced4b723aa7e8a5a55881774d1fd6e8a86e702399e7008b5281e1384ecd"}
Oct 30 18:54:05.744381 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:05.744349    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:05.962650 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:05.962621    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb25626-pr6sm"
Oct 30 18:54:05.992683 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:05.992649    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8b2909a-6460-47d2-99f2-7fe93b21e81c-kube-api-access-d6vmx\" (UniqueName: \"kubernetes.io/projected/e8b2909a-6460-47d2-99f2-7fe93b21e81c-kube-api-access-d6vmx\") pod \"e8b2909a-6460-47d2-99f2-7fe93b21e81c\" (UID: \"e8b2909a-6460-47d2-99f2-7fe93b21e81c\") "
Oct 30 18:54:05.992818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:05.992726    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8b2909a-6460-47d2-99f2-7fe93b21e81c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8b2909a-6460-47d2-99f2-7fe93b21e81c-maas-api-openshift-ca\") pod \"e8b2909a-6460-47d2-99f2-7fe93b21e81c\" (UID: \"e8b2909a-6460-47d2-99f2-7fe93b21e81c\") "
Oct 30 18:54:05.992818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:05.992758    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8b2909a-6460-47d2-99f2-7fe93b21e81c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8b2909a-6460-47d2-99f2-7fe93b21e81c-maas-api-xks-ca\") pod \"e8b2909a-6460-47d2-99f2-7fe93b21e81c\" (UID: \"e8b2909a-6460-47d2-99f2-7fe93b21e81c\") "
Oct 30 18:54:05.993047 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:05.993024    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8b2909a-6460-47d2-99f2-7fe93b21e81c-maas-api-openshift-ca" pod "e8b2909a-6460-47d2-99f2-7fe93b21e81c" (UID: "e8b2909a-6460-47d2-99f2-7fe93b21e81c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:05.993159 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:05.993145    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8b2909a-6460-47d2-99f2-7fe93b21e81c-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:05.994783 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:05.994723    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8b2909a-6460-47d2-99f2-7fe93b21e81c-maas-api-xks-ca" pod "e8b2909a-6460-47d2-99f2-7fe93b21e81c" (UID: "e8b2909a-6460-47d2-99f2-7fe93b21e81c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:54:05.994783 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:05.994745    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8b2909a-6460-47d2-99f2-7fe93b21e81c-kube-api-access-d6vmx" pod "e8b2909a-6460-47d2-99f2-7fe93b21e81c" (UID: "e8b2909a-6460-47d2-99f2-7fe93b21e81c"). InnerVolumeSpecName "kube-api-access-d6vmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:06.094385 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:06.094353    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6vmx\" (UniqueName: \"kubernetes.io/projected/e8b2909a-6460-47d2-99f2-7fe93b21e81c-kube-api-access-d6vmx\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:06.094385 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:06.094385    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8b2909a-6460-47d2-99f2-7fe93b21e81c-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:06.129197 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:06.129050    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:06.813356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:06.813326    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb25626-pr6sm"
Oct 30 18:54:06.813557 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:06.813332    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcb25626-pr6sm" event={"ID":"e8b2909a-6460-47d2-99f2-7fe93b21e81c","Type":"ContainerDied","Data":"a8a09fed837e2a1ba63145527fcebe010d376f269e7d13d08f23f1b471171ea3"}
Oct 30 18:54:06.813557 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:06.813440    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8a09fed837e2a1ba63145527fcebe010d376f269e7d13d08f23f1b471171ea3"
Oct 30 18:54:11.011850 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:11.011796    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:13.802674 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:13.802634    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37c03f3a-csvn4"]
Oct 30 18:54:13.803388 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:13.803368    4924 cpu_manager.go:414] "containerMap: removing container" podUID="e8b2909a-6460-47d2-99f2-7fe93b21e81c" containerName="revoke-keys"
Oct 30 18:54:13.803489 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:13.803389    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8b2909a-6460-47d2-99f2-7fe93b21e81c" containerName="revoke-keys"
Oct 30 18:54:13.803575 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:13.803562    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8b2909a-6460-47d2-99f2-7fe93b21e81c" containerName="revoke-keys"
Oct 30 18:54:13.807273 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:13.807253    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c03f3a-csvn4"
Oct 30 18:54:13.861147 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:13.860992    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37c03f3a-csvn4"]
Oct 30 18:54:13.881431 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:13.881397    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54c3946-3c6c-4aab-876f-f0f8c2817604-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37c03f3a-csvn4\" (UID: \"f54c3946-3c6c-4aab-876f-f0f8c2817604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c03f3a-csvn4"
Oct 30 18:54:13.881581 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:13.881527    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54c3946-3c6c-4aab-876f-f0f8c2817604-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37c03f3a-csvn4\" (UID: \"f54c3946-3c6c-4aab-876f-f0f8c2817604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c03f3a-csvn4"
Oct 30 18:54:13.881640 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:13.881599    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk6v2\" (UniqueName: \"kubernetes.io/projected/f54c3946-3c6c-4aab-876f-f0f8c2817604-kube-api-access-bk6v2\") pod \"maas-api-revoke-sub-37c03f3a-csvn4\" (UID: \"f54c3946-3c6c-4aab-876f-f0f8c2817604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c03f3a-csvn4"
Oct 30 18:54:13.903627 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:13.903601    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:54:13.903795 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:13.903626    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:54:13.903795 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:13.903632    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:54:13.903795 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:13.903716    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 18:54:13.983371 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:13.983340    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54c3946-3c6c-4aab-876f-f0f8c2817604-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37c03f3a-csvn4\" (UID: \"f54c3946-3c6c-4aab-876f-f0f8c2817604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c03f3a-csvn4"
Oct 30 18:54:13.983515 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:13.983390    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54c3946-3c6c-4aab-876f-f0f8c2817604-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37c03f3a-csvn4\" (UID: \"f54c3946-3c6c-4aab-876f-f0f8c2817604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c03f3a-csvn4"
Oct 30 18:54:13.983515 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:13.983421    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk6v2\" (UniqueName: \"kubernetes.io/projected/f54c3946-3c6c-4aab-876f-f0f8c2817604-kube-api-access-bk6v2\") pod \"maas-api-revoke-sub-37c03f3a-csvn4\" (UID: \"f54c3946-3c6c-4aab-876f-f0f8c2817604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c03f3a-csvn4"
Oct 30 18:54:13.984016 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:13.983997    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54c3946-3c6c-4aab-876f-f0f8c2817604-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37c03f3a-csvn4\" (UID: \"f54c3946-3c6c-4aab-876f-f0f8c2817604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c03f3a-csvn4"
Oct 30 18:54:13.985650 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:13.985633    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54c3946-3c6c-4aab-876f-f0f8c2817604-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37c03f3a-csvn4\" (UID: \"f54c3946-3c6c-4aab-876f-f0f8c2817604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c03f3a-csvn4"
Oct 30 18:54:14.037465 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:14.037437    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk6v2\" (UniqueName: \"kubernetes.io/projected/f54c3946-3c6c-4aab-876f-f0f8c2817604-kube-api-access-bk6v2\") pod \"maas-api-revoke-sub-37c03f3a-csvn4\" (UID: \"f54c3946-3c6c-4aab-876f-f0f8c2817604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c03f3a-csvn4"
Oct 30 18:54:14.120405 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:14.120378    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c03f3a-csvn4"
Oct 30 18:54:14.232748 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:14.232653    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:14.496334 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:14.496251    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37c03f3a-csvn4"]
Oct 30 18:54:14.499845 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:54:14.499815    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54c3946_3c6c_4aab_876f_f0f8c2817604.slice/crio-d91cd1289f8e02a5a1ddf259eb612fc7fb3e593e553fbd4e16b10e905c33a0ef WatchSource:0}: Error finding container d91cd1289f8e02a5a1ddf259eb612fc7fb3e593e553fbd4e16b10e905c33a0ef: Status 404 returned error can't find the container with id d91cd1289f8e02a5a1ddf259eb612fc7fb3e593e553fbd4e16b10e905c33a0ef
Oct 30 18:54:14.870930 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:14.870895    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c03f3a-csvn4" event={"ID":"f54c3946-3c6c-4aab-876f-f0f8c2817604","Type":"ContainerStarted","Data":"1822e2665235bd60a5d1884412a098c4ebc3f021f5d44ab59baadf00a2df8dad"}
Oct 30 18:54:14.871386 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:14.870944    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c03f3a-csvn4" event={"ID":"f54c3946-3c6c-4aab-876f-f0f8c2817604","Type":"ContainerStarted","Data":"d91cd1289f8e02a5a1ddf259eb612fc7fb3e593e553fbd4e16b10e905c33a0ef"}
Oct 30 18:54:14.978775 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:14.978728    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c03f3a-csvn4" podStartSLOduration=1.978715123 podStartE2EDuration="1.978715123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:14.978262762 +0000 UTC m=+1056.255313241" watchObservedRunningTime="2026-10-30 18:54:14.978715123 +0000 UTC m=+1056.255765567"
Oct 30 18:54:15.710748 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:54:15.710670    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:54:15.888447 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:15.888407    4924 generic.go:356] "Generic (PLEG): container finished" podID="f54c3946-3c6c-4aab-876f-f0f8c2817604" containerID="1822e2665235bd60a5d1884412a098c4ebc3f021f5d44ab59baadf00a2df8dad" exitCode=0
Oct 30 18:54:15.888881 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:15.888535    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c03f3a-csvn4" event={"ID":"f54c3946-3c6c-4aab-876f-f0f8c2817604","Type":"ContainerDied","Data":"1822e2665235bd60a5d1884412a098c4ebc3f021f5d44ab59baadf00a2df8dad"}
Oct 30 18:54:15.974327 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:15.974194    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:17.043799 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:17.043776    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c03f3a-csvn4"
Oct 30 18:54:17.124160 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:17.124132    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f54c3946-3c6c-4aab-876f-f0f8c2817604-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54c3946-3c6c-4aab-876f-f0f8c2817604-maas-api-openshift-ca\") pod \"f54c3946-3c6c-4aab-876f-f0f8c2817604\" (UID: \"f54c3946-3c6c-4aab-876f-f0f8c2817604\") "
Oct 30 18:54:17.124336 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:17.124242    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f54c3946-3c6c-4aab-876f-f0f8c2817604-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54c3946-3c6c-4aab-876f-f0f8c2817604-maas-api-xks-ca\") pod \"f54c3946-3c6c-4aab-876f-f0f8c2817604\" (UID: \"f54c3946-3c6c-4aab-876f-f0f8c2817604\") "
Oct 30 18:54:17.124410 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:17.124337    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f54c3946-3c6c-4aab-876f-f0f8c2817604-kube-api-access-bk6v2\" (UniqueName: \"kubernetes.io/projected/f54c3946-3c6c-4aab-876f-f0f8c2817604-kube-api-access-bk6v2\") pod \"f54c3946-3c6c-4aab-876f-f0f8c2817604\" (UID: \"f54c3946-3c6c-4aab-876f-f0f8c2817604\") "
Oct 30 18:54:17.124558 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:17.124534    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f54c3946-3c6c-4aab-876f-f0f8c2817604-maas-api-openshift-ca" pod "f54c3946-3c6c-4aab-876f-f0f8c2817604" (UID: "f54c3946-3c6c-4aab-876f-f0f8c2817604"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:17.127428 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:17.127384    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54c3946-3c6c-4aab-876f-f0f8c2817604-kube-api-access-bk6v2" pod "f54c3946-3c6c-4aab-876f-f0f8c2817604" (UID: "f54c3946-3c6c-4aab-876f-f0f8c2817604"). InnerVolumeSpecName "kube-api-access-bk6v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:17.135253 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:17.135225    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54c3946-3c6c-4aab-876f-f0f8c2817604-maas-api-xks-ca" pod "f54c3946-3c6c-4aab-876f-f0f8c2817604" (UID: "f54c3946-3c6c-4aab-876f-f0f8c2817604"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:54:17.225847 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:17.225816    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54c3946-3c6c-4aab-876f-f0f8c2817604-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.225847 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:17.225845    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk6v2\" (UniqueName: \"kubernetes.io/projected/f54c3946-3c6c-4aab-876f-f0f8c2817604-kube-api-access-bk6v2\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.226040 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:17.225856    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54c3946-3c6c-4aab-876f-f0f8c2817604-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:17.900557 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:17.900525    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c03f3a-csvn4"
Oct 30 18:54:17.900719 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:17.900555    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c03f3a-csvn4" event={"ID":"f54c3946-3c6c-4aab-876f-f0f8c2817604","Type":"ContainerDied","Data":"d91cd1289f8e02a5a1ddf259eb612fc7fb3e593e553fbd4e16b10e905c33a0ef"}
Oct 30 18:54:17.900719 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:17.900606    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d91cd1289f8e02a5a1ddf259eb612fc7fb3e593e553fbd4e16b10e905c33a0ef"
Oct 30 18:54:18.908297 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:18.908256    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:19.100128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:19.100080    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:20.919514 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:20.919468    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:24.563090 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:24.551653    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72842ed0-ndk5l"]
Oct 30 18:54:24.565276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:24.565248    4924 cpu_manager.go:414] "containerMap: removing container" podUID="f54c3946-3c6c-4aab-876f-f0f8c2817604" containerName="revoke-keys"
Oct 30 18:54:24.574135 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:24.574109    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f54c3946-3c6c-4aab-876f-f0f8c2817604" containerName="revoke-keys"
Oct 30 18:54:24.574483 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:24.574470    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="f54c3946-3c6c-4aab-876f-f0f8c2817604" containerName="revoke-keys"
Oct 30 18:54:24.578996 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:24.578981    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72842ed0-ndk5l"
Oct 30 18:54:24.586422 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:24.586401    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72842ed0-ndk5l"]
Oct 30 18:54:24.678421 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:24.678383    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrrnm\" (UniqueName: \"kubernetes.io/projected/e7d023c1-3461-491d-b3fa-3f3a07f01e7b-kube-api-access-qrrnm\") pod \"maas-api-revoke-sub-72842ed0-ndk5l\" (UID: \"e7d023c1-3461-491d-b3fa-3f3a07f01e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72842ed0-ndk5l"
Oct 30 18:54:24.678707 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:24.678691    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7d023c1-3461-491d-b3fa-3f3a07f01e7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72842ed0-ndk5l\" (UID: \"e7d023c1-3461-491d-b3fa-3f3a07f01e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72842ed0-ndk5l"
Oct 30 18:54:24.678904 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:24.678893    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7d023c1-3461-491d-b3fa-3f3a07f01e7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72842ed0-ndk5l\" (UID: \"e7d023c1-3461-491d-b3fa-3f3a07f01e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72842ed0-ndk5l"
Oct 30 18:54:24.683137 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:24.683115    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:54:24.683286 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:24.683270    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:54:24.683467 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:24.683453    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 18:54:24.683638 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:24.683628    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:54:24.779831 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:24.779799    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7d023c1-3461-491d-b3fa-3f3a07f01e7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72842ed0-ndk5l\" (UID: \"e7d023c1-3461-491d-b3fa-3f3a07f01e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72842ed0-ndk5l"
Oct 30 18:54:24.780036 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:24.779894    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7d023c1-3461-491d-b3fa-3f3a07f01e7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72842ed0-ndk5l\" (UID: \"e7d023c1-3461-491d-b3fa-3f3a07f01e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72842ed0-ndk5l"
Oct 30 18:54:24.780036 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:24.779928    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrrnm\" (UniqueName: \"kubernetes.io/projected/e7d023c1-3461-491d-b3fa-3f3a07f01e7b-kube-api-access-qrrnm\") pod \"maas-api-revoke-sub-72842ed0-ndk5l\" (UID: \"e7d023c1-3461-491d-b3fa-3f3a07f01e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72842ed0-ndk5l"
Oct 30 18:54:24.780468 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:24.780448    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7d023c1-3461-491d-b3fa-3f3a07f01e7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72842ed0-ndk5l\" (UID: \"e7d023c1-3461-491d-b3fa-3f3a07f01e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72842ed0-ndk5l"
Oct 30 18:54:24.782229 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:24.782212    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7d023c1-3461-491d-b3fa-3f3a07f01e7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72842ed0-ndk5l\" (UID: \"e7d023c1-3461-491d-b3fa-3f3a07f01e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72842ed0-ndk5l"
Oct 30 18:54:24.834497 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:24.834434    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrrnm\" (UniqueName: \"kubernetes.io/projected/e7d023c1-3461-491d-b3fa-3f3a07f01e7b-kube-api-access-qrrnm\") pod \"maas-api-revoke-sub-72842ed0-ndk5l\" (UID: \"e7d023c1-3461-491d-b3fa-3f3a07f01e7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72842ed0-ndk5l"
Oct 30 18:54:24.895472 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:24.895444    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72842ed0-ndk5l"
Oct 30 18:54:25.267267 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:25.267244    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72842ed0-ndk5l"]
Oct 30 18:54:25.269018 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:54:25.268923    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d023c1_3461_491d_b3fa_3f3a07f01e7b.slice/crio-03bac8783ab424de8bae6e61e1f7436d17d1542860dc3c17cff1c36c60b0ce4a WatchSource:0}: Error finding container 03bac8783ab424de8bae6e61e1f7436d17d1542860dc3c17cff1c36c60b0ce4a: Status 404 returned error can't find the container with id 03bac8783ab424de8bae6e61e1f7436d17d1542860dc3c17cff1c36c60b0ce4a
Oct 30 18:54:25.823701 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:25.823672    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:25.955923 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:25.955882    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72842ed0-ndk5l" event={"ID":"e7d023c1-3461-491d-b3fa-3f3a07f01e7b","Type":"ContainerStarted","Data":"b4002e1d7184303f39a20d3df227fae164d4cf8db400ed8faeb6bbc6f3c4e8a3"}
Oct 30 18:54:25.956155 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:25.955946    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72842ed0-ndk5l" event={"ID":"e7d023c1-3461-491d-b3fa-3f3a07f01e7b","Type":"ContainerStarted","Data":"03bac8783ab424de8bae6e61e1f7436d17d1542860dc3c17cff1c36c60b0ce4a"}
Oct 30 18:54:26.069649 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:26.069603    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72842ed0-ndk5l" podStartSLOduration=2.069591519 podStartE2EDuration="2.069591519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:26.067796244 +0000 UTC m=+1067.344846689" watchObservedRunningTime="2026-10-30 18:54:26.069591519 +0000 UTC m=+1067.346641965"
Oct 30 18:54:26.962742 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:26.962703    4924 generic.go:356] "Generic (PLEG): container finished" podID="e7d023c1-3461-491d-b3fa-3f3a07f01e7b" containerID="b4002e1d7184303f39a20d3df227fae164d4cf8db400ed8faeb6bbc6f3c4e8a3" exitCode=0
Oct 30 18:54:26.963154 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:26.962763    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72842ed0-ndk5l" event={"ID":"e7d023c1-3461-491d-b3fa-3f3a07f01e7b","Type":"ContainerDied","Data":"b4002e1d7184303f39a20d3df227fae164d4cf8db400ed8faeb6bbc6f3c4e8a3"}
Oct 30 18:54:28.113375 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:28.113352    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72842ed0-ndk5l"
Oct 30 18:54:28.211545 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:28.211516    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7d023c1-3461-491d-b3fa-3f3a07f01e7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7d023c1-3461-491d-b3fa-3f3a07f01e7b-maas-api-openshift-ca\") pod \"e7d023c1-3461-491d-b3fa-3f3a07f01e7b\" (UID: \"e7d023c1-3461-491d-b3fa-3f3a07f01e7b\") "
Oct 30 18:54:28.211725 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:28.211619    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7d023c1-3461-491d-b3fa-3f3a07f01e7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7d023c1-3461-491d-b3fa-3f3a07f01e7b-maas-api-xks-ca\") pod \"e7d023c1-3461-491d-b3fa-3f3a07f01e7b\" (UID: \"e7d023c1-3461-491d-b3fa-3f3a07f01e7b\") "
Oct 30 18:54:28.211725 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:28.211653    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7d023c1-3461-491d-b3fa-3f3a07f01e7b-kube-api-access-qrrnm\" (UniqueName: \"kubernetes.io/projected/e7d023c1-3461-491d-b3fa-3f3a07f01e7b-kube-api-access-qrrnm\") pod \"e7d023c1-3461-491d-b3fa-3f3a07f01e7b\" (UID: \"e7d023c1-3461-491d-b3fa-3f3a07f01e7b\") "
Oct 30 18:54:28.211905 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:28.211878    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7d023c1-3461-491d-b3fa-3f3a07f01e7b-maas-api-openshift-ca" pod "e7d023c1-3461-491d-b3fa-3f3a07f01e7b" (UID: "e7d023c1-3461-491d-b3fa-3f3a07f01e7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:28.213594 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:28.213569    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7d023c1-3461-491d-b3fa-3f3a07f01e7b-maas-api-xks-ca" pod "e7d023c1-3461-491d-b3fa-3f3a07f01e7b" (UID: "e7d023c1-3461-491d-b3fa-3f3a07f01e7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:54:28.213688 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:28.213656    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d023c1-3461-491d-b3fa-3f3a07f01e7b-kube-api-access-qrrnm" pod "e7d023c1-3461-491d-b3fa-3f3a07f01e7b" (UID: "e7d023c1-3461-491d-b3fa-3f3a07f01e7b"). InnerVolumeSpecName "kube-api-access-qrrnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:28.313130 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:28.313092    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrrnm\" (UniqueName: \"kubernetes.io/projected/e7d023c1-3461-491d-b3fa-3f3a07f01e7b-kube-api-access-qrrnm\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:28.313130 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:28.313124    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7d023c1-3461-491d-b3fa-3f3a07f01e7b-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:28.313130 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:28.313139    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7d023c1-3461-491d-b3fa-3f3a07f01e7b-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:28.975805 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:28.975773    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72842ed0-ndk5l"
Oct 30 18:54:28.975978 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:28.975807    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72842ed0-ndk5l" event={"ID":"e7d023c1-3461-491d-b3fa-3f3a07f01e7b","Type":"ContainerDied","Data":"03bac8783ab424de8bae6e61e1f7436d17d1542860dc3c17cff1c36c60b0ce4a"}
Oct 30 18:54:28.975978 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:28.975849    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03bac8783ab424de8bae6e61e1f7436d17d1542860dc3c17cff1c36c60b0ce4a"
Oct 30 18:54:30.820326 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:30.820292    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:34.244316 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:34.244281    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:35.286616 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:35.286584    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5be1def-gm8zk"]
Oct 30 18:54:35.287410 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:35.287391    4924 cpu_manager.go:414] "containerMap: removing container" podUID="e7d023c1-3461-491d-b3fa-3f3a07f01e7b" containerName="revoke-keys"
Oct 30 18:54:35.287461 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:35.287413    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7d023c1-3461-491d-b3fa-3f3a07f01e7b" containerName="revoke-keys"
Oct 30 18:54:35.287564 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:35.287553    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7d023c1-3461-491d-b3fa-3f3a07f01e7b" containerName="revoke-keys"
Oct 30 18:54:35.292346 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:35.292330    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5be1def-gm8zk"]
Oct 30 18:54:35.292423 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:35.292413    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5be1def-gm8zk"
Oct 30 18:54:35.390430 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:35.390395    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:54:35.390591 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:35.390444    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:54:35.390795 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:35.390778    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:54:35.390837 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:35.390809    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 18:54:35.400238 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:35.400204    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlnb8\" (UniqueName: \"kubernetes.io/projected/51274534-a4ac-4b78-9419-707c67528744-kube-api-access-rlnb8\") pod \"maas-api-revoke-sub-f5be1def-gm8zk\" (UID: \"51274534-a4ac-4b78-9419-707c67528744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5be1def-gm8zk"
Oct 30 18:54:35.400238 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:35.400233    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51274534-a4ac-4b78-9419-707c67528744-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5be1def-gm8zk\" (UID: \"51274534-a4ac-4b78-9419-707c67528744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5be1def-gm8zk"
Oct 30 18:54:35.400445 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:35.400329    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51274534-a4ac-4b78-9419-707c67528744-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5be1def-gm8zk\" (UID: \"51274534-a4ac-4b78-9419-707c67528744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5be1def-gm8zk"
Oct 30 18:54:35.501339 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:35.501305    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlnb8\" (UniqueName: \"kubernetes.io/projected/51274534-a4ac-4b78-9419-707c67528744-kube-api-access-rlnb8\") pod \"maas-api-revoke-sub-f5be1def-gm8zk\" (UID: \"51274534-a4ac-4b78-9419-707c67528744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5be1def-gm8zk"
Oct 30 18:54:35.501339 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:35.501342    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51274534-a4ac-4b78-9419-707c67528744-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5be1def-gm8zk\" (UID: \"51274534-a4ac-4b78-9419-707c67528744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5be1def-gm8zk"
Oct 30 18:54:35.501542 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:35.501397    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51274534-a4ac-4b78-9419-707c67528744-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5be1def-gm8zk\" (UID: \"51274534-a4ac-4b78-9419-707c67528744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5be1def-gm8zk"
Oct 30 18:54:35.502135 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:35.502117    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51274534-a4ac-4b78-9419-707c67528744-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5be1def-gm8zk\" (UID: \"51274534-a4ac-4b78-9419-707c67528744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5be1def-gm8zk"
Oct 30 18:54:35.503599 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:35.503581    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51274534-a4ac-4b78-9419-707c67528744-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5be1def-gm8zk\" (UID: \"51274534-a4ac-4b78-9419-707c67528744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5be1def-gm8zk"
Oct 30 18:54:35.560683 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:35.560569    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlnb8\" (UniqueName: \"kubernetes.io/projected/51274534-a4ac-4b78-9419-707c67528744-kube-api-access-rlnb8\") pod \"maas-api-revoke-sub-f5be1def-gm8zk\" (UID: \"51274534-a4ac-4b78-9419-707c67528744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5be1def-gm8zk"
Oct 30 18:54:35.605581 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:35.605548    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5be1def-gm8zk"
Oct 30 18:54:35.827210 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:35.827125    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:35.995763 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:54:35.995723    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51274534_a4ac_4b78_9419_707c67528744.slice/crio-577ecfff143c4f05ad3fe7d90d85e7b3a340ef685b63df1a0e1c0a814bbaaa8c WatchSource:0}: Error finding container 577ecfff143c4f05ad3fe7d90d85e7b3a340ef685b63df1a0e1c0a814bbaaa8c: Status 404 returned error can't find the container with id 577ecfff143c4f05ad3fe7d90d85e7b3a340ef685b63df1a0e1c0a814bbaaa8c
Oct 30 18:54:35.996522 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:35.996492    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5be1def-gm8zk"]
Oct 30 18:54:36.026517 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:36.026472    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5be1def-gm8zk" event={"ID":"51274534-a4ac-4b78-9419-707c67528744","Type":"ContainerStarted","Data":"577ecfff143c4f05ad3fe7d90d85e7b3a340ef685b63df1a0e1c0a814bbaaa8c"}
Oct 30 18:54:37.033851 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:37.033811    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5be1def-gm8zk" event={"ID":"51274534-a4ac-4b78-9419-707c67528744","Type":"ContainerStarted","Data":"67e40f64d31558d189f86f1ac9fa5464e6de924e3a44b8e54b9105a6439237e1"}
Oct 30 18:54:37.192494 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:37.192445    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5be1def-gm8zk" podStartSLOduration=2.192432041 podStartE2EDuration="2.192432041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:37.144340896 +0000 UTC m=+1078.421391342" watchObservedRunningTime="2026-10-30 18:54:37.192432041 +0000 UTC m=+1078.469482486"
Oct 30 18:54:37.239641 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:37.239600    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f478c53e-wsxp2"]
Oct 30 18:54:37.243663 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:37.243642    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f478c53e-wsxp2"
Oct 30 18:54:37.299427 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:37.299358    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f478c53e-wsxp2"]
Oct 30 18:54:37.330848 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:37.330820    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c96a23-72c8-44c6-8d2b-62d85d750bb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f478c53e-wsxp2\" (UID: \"26c96a23-72c8-44c6-8d2b-62d85d750bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f478c53e-wsxp2"
Oct 30 18:54:37.331011 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:37.330973    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bntjp\" (UniqueName: \"kubernetes.io/projected/26c96a23-72c8-44c6-8d2b-62d85d750bb4-kube-api-access-bntjp\") pod \"maas-api-revoke-sub-f478c53e-wsxp2\" (UID: \"26c96a23-72c8-44c6-8d2b-62d85d750bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f478c53e-wsxp2"
Oct 30 18:54:37.331079 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:37.331019    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c96a23-72c8-44c6-8d2b-62d85d750bb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f478c53e-wsxp2\" (UID: \"26c96a23-72c8-44c6-8d2b-62d85d750bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f478c53e-wsxp2"
Oct 30 18:54:37.432153 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:37.432112    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bntjp\" (UniqueName: \"kubernetes.io/projected/26c96a23-72c8-44c6-8d2b-62d85d750bb4-kube-api-access-bntjp\") pod \"maas-api-revoke-sub-f478c53e-wsxp2\" (UID: \"26c96a23-72c8-44c6-8d2b-62d85d750bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f478c53e-wsxp2"
Oct 30 18:54:37.432341 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:37.432179    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c96a23-72c8-44c6-8d2b-62d85d750bb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f478c53e-wsxp2\" (UID: \"26c96a23-72c8-44c6-8d2b-62d85d750bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f478c53e-wsxp2"
Oct 30 18:54:37.432341 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:37.432309    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c96a23-72c8-44c6-8d2b-62d85d750bb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f478c53e-wsxp2\" (UID: \"26c96a23-72c8-44c6-8d2b-62d85d750bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f478c53e-wsxp2"
Oct 30 18:54:37.432832 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:37.432812    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c96a23-72c8-44c6-8d2b-62d85d750bb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f478c53e-wsxp2\" (UID: \"26c96a23-72c8-44c6-8d2b-62d85d750bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f478c53e-wsxp2"
Oct 30 18:54:37.434591 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:37.434561    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c96a23-72c8-44c6-8d2b-62d85d750bb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f478c53e-wsxp2\" (UID: \"26c96a23-72c8-44c6-8d2b-62d85d750bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f478c53e-wsxp2"
Oct 30 18:54:37.487661 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:37.487632    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bntjp\" (UniqueName: \"kubernetes.io/projected/26c96a23-72c8-44c6-8d2b-62d85d750bb4-kube-api-access-bntjp\") pod \"maas-api-revoke-sub-f478c53e-wsxp2\" (UID: \"26c96a23-72c8-44c6-8d2b-62d85d750bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f478c53e-wsxp2"
Oct 30 18:54:37.557178 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:37.557108    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f478c53e-wsxp2"
Oct 30 18:54:37.932081 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:37.932039    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f478c53e-wsxp2"]
Oct 30 18:54:37.934200 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:54:37.934175    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c96a23_72c8_44c6_8d2b_62d85d750bb4.slice/crio-2c688ef29e73a462a33baba0ad4851ae257039947e5b59a43d2995489e282f7f WatchSource:0}: Error finding container 2c688ef29e73a462a33baba0ad4851ae257039947e5b59a43d2995489e282f7f: Status 404 returned error can't find the container with id 2c688ef29e73a462a33baba0ad4851ae257039947e5b59a43d2995489e282f7f
Oct 30 18:54:38.041098 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:38.041047    4924 generic.go:356] "Generic (PLEG): container finished" podID="51274534-a4ac-4b78-9419-707c67528744" containerID="67e40f64d31558d189f86f1ac9fa5464e6de924e3a44b8e54b9105a6439237e1" exitCode=0
Oct 30 18:54:38.041455 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:38.041132    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5be1def-gm8zk" event={"ID":"51274534-a4ac-4b78-9419-707c67528744","Type":"ContainerDied","Data":"67e40f64d31558d189f86f1ac9fa5464e6de924e3a44b8e54b9105a6439237e1"}
Oct 30 18:54:38.042492 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:38.042464    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f478c53e-wsxp2" event={"ID":"26c96a23-72c8-44c6-8d2b-62d85d750bb4","Type":"ContainerStarted","Data":"2c688ef29e73a462a33baba0ad4851ae257039947e5b59a43d2995489e282f7f"}
Oct 30 18:54:39.049533 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.049455    4924 generic.go:356] "Generic (PLEG): container finished" podID="26c96a23-72c8-44c6-8d2b-62d85d750bb4" containerID="e784dd833e5ea8326da0b38950c04d2c4c50ae8006c9da704b3208176b2b556c" exitCode=0
Oct 30 18:54:39.049533 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.049509    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f478c53e-wsxp2" event={"ID":"26c96a23-72c8-44c6-8d2b-62d85d750bb4","Type":"ContainerDied","Data":"e784dd833e5ea8326da0b38950c04d2c4c50ae8006c9da704b3208176b2b556c"}
Oct 30 18:54:39.193958 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.193932    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5be1def-gm8zk"
Oct 30 18:54:39.251044 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.251007    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51274534-a4ac-4b78-9419-707c67528744-kube-api-access-rlnb8\" (UniqueName: \"kubernetes.io/projected/51274534-a4ac-4b78-9419-707c67528744-kube-api-access-rlnb8\") pod \"51274534-a4ac-4b78-9419-707c67528744\" (UID: \"51274534-a4ac-4b78-9419-707c67528744\") "
Oct 30 18:54:39.251188 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.251122    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51274534-a4ac-4b78-9419-707c67528744-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51274534-a4ac-4b78-9419-707c67528744-maas-api-openshift-ca\") pod \"51274534-a4ac-4b78-9419-707c67528744\" (UID: \"51274534-a4ac-4b78-9419-707c67528744\") "
Oct 30 18:54:39.251188 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.251161    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51274534-a4ac-4b78-9419-707c67528744-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51274534-a4ac-4b78-9419-707c67528744-maas-api-xks-ca\") pod \"51274534-a4ac-4b78-9419-707c67528744\" (UID: \"51274534-a4ac-4b78-9419-707c67528744\") "
Oct 30 18:54:39.252303 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.252274    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51274534-a4ac-4b78-9419-707c67528744-maas-api-openshift-ca" pod "51274534-a4ac-4b78-9419-707c67528744" (UID: "51274534-a4ac-4b78-9419-707c67528744"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:39.254156 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.254128    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51274534-a4ac-4b78-9419-707c67528744-maas-api-xks-ca" pod "51274534-a4ac-4b78-9419-707c67528744" (UID: "51274534-a4ac-4b78-9419-707c67528744"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:54:39.254691 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.254665    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51274534-a4ac-4b78-9419-707c67528744-kube-api-access-rlnb8" pod "51274534-a4ac-4b78-9419-707c67528744" (UID: "51274534-a4ac-4b78-9419-707c67528744"). InnerVolumeSpecName "kube-api-access-rlnb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:39.352843 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.352810    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlnb8\" (UniqueName: \"kubernetes.io/projected/51274534-a4ac-4b78-9419-707c67528744-kube-api-access-rlnb8\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:39.352843 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.352839    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51274534-a4ac-4b78-9419-707c67528744-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:39.353027 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.352856    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51274534-a4ac-4b78-9419-707c67528744-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:39.672364 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.672284    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dfda3bc-9krtl"]
Oct 30 18:54:39.672935 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.672910    4924 cpu_manager.go:414] "containerMap: removing container" podUID="51274534-a4ac-4b78-9419-707c67528744" containerName="revoke-keys"
Oct 30 18:54:39.672935 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.672932    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51274534-a4ac-4b78-9419-707c67528744" containerName="revoke-keys"
Oct 30 18:54:39.673140 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.673092    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="51274534-a4ac-4b78-9419-707c67528744" containerName="revoke-keys"
Oct 30 18:54:39.676504 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.676487    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dfda3bc-9krtl"
Oct 30 18:54:39.730607 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.730579    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dfda3bc-9krtl"]
Oct 30 18:54:39.757756 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.757713    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a0e3b75-0597-4169-9dfa-848c4359eec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dfda3bc-9krtl\" (UID: \"1a0e3b75-0597-4169-9dfa-848c4359eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dfda3bc-9krtl"
Oct 30 18:54:39.757882 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.757770    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2zzf\" (UniqueName: \"kubernetes.io/projected/1a0e3b75-0597-4169-9dfa-848c4359eec3-kube-api-access-r2zzf\") pod \"maas-api-revoke-sub-3dfda3bc-9krtl\" (UID: \"1a0e3b75-0597-4169-9dfa-848c4359eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dfda3bc-9krtl"
Oct 30 18:54:39.757952 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.757935    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a0e3b75-0597-4169-9dfa-848c4359eec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dfda3bc-9krtl\" (UID: \"1a0e3b75-0597-4169-9dfa-848c4359eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dfda3bc-9krtl"
Oct 30 18:54:39.859072 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.859031    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a0e3b75-0597-4169-9dfa-848c4359eec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dfda3bc-9krtl\" (UID: \"1a0e3b75-0597-4169-9dfa-848c4359eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dfda3bc-9krtl"
Oct 30 18:54:39.859223 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.859147    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a0e3b75-0597-4169-9dfa-848c4359eec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dfda3bc-9krtl\" (UID: \"1a0e3b75-0597-4169-9dfa-848c4359eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dfda3bc-9krtl"
Oct 30 18:54:39.859223 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.859176    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2zzf\" (UniqueName: \"kubernetes.io/projected/1a0e3b75-0597-4169-9dfa-848c4359eec3-kube-api-access-r2zzf\") pod \"maas-api-revoke-sub-3dfda3bc-9krtl\" (UID: \"1a0e3b75-0597-4169-9dfa-848c4359eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dfda3bc-9krtl"
Oct 30 18:54:39.859802 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.859782    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a0e3b75-0597-4169-9dfa-848c4359eec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dfda3bc-9krtl\" (UID: \"1a0e3b75-0597-4169-9dfa-848c4359eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dfda3bc-9krtl"
Oct 30 18:54:39.861361 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.861340    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a0e3b75-0597-4169-9dfa-848c4359eec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dfda3bc-9krtl\" (UID: \"1a0e3b75-0597-4169-9dfa-848c4359eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dfda3bc-9krtl"
Oct 30 18:54:39.914160 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.914032    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2zzf\" (UniqueName: \"kubernetes.io/projected/1a0e3b75-0597-4169-9dfa-848c4359eec3-kube-api-access-r2zzf\") pod \"maas-api-revoke-sub-3dfda3bc-9krtl\" (UID: \"1a0e3b75-0597-4169-9dfa-848c4359eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dfda3bc-9krtl"
Oct 30 18:54:39.990325 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:39.990259    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dfda3bc-9krtl"
Oct 30 18:54:40.063793 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.063761    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5be1def-gm8zk"
Oct 30 18:54:40.065162 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.065127    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5be1def-gm8zk" event={"ID":"51274534-a4ac-4b78-9419-707c67528744","Type":"ContainerDied","Data":"577ecfff143c4f05ad3fe7d90d85e7b3a340ef685b63df1a0e1c0a814bbaaa8c"}
Oct 30 18:54:40.065286 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.065191    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="577ecfff143c4f05ad3fe7d90d85e7b3a340ef685b63df1a0e1c0a814bbaaa8c"
Oct 30 18:54:40.234281 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.234259    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f478c53e-wsxp2"
Oct 30 18:54:40.363852 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:54:40.363823    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a0e3b75_0597_4169_9dfa_848c4359eec3.slice/crio-0de0546c5e6bb8ff5c4bde018705170259c04ab306a64aec62d8125d2f69a3a8 WatchSource:0}: Error finding container 0de0546c5e6bb8ff5c4bde018705170259c04ab306a64aec62d8125d2f69a3a8: Status 404 returned error can't find the container with id 0de0546c5e6bb8ff5c4bde018705170259c04ab306a64aec62d8125d2f69a3a8
Oct 30 18:54:40.365226 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.365202    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26c96a23-72c8-44c6-8d2b-62d85d750bb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c96a23-72c8-44c6-8d2b-62d85d750bb4-maas-api-xks-ca\") pod \"26c96a23-72c8-44c6-8d2b-62d85d750bb4\" (UID: \"26c96a23-72c8-44c6-8d2b-62d85d750bb4\") "
Oct 30 18:54:40.365430 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.365405    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26c96a23-72c8-44c6-8d2b-62d85d750bb4-kube-api-access-bntjp\" (UniqueName: \"kubernetes.io/projected/26c96a23-72c8-44c6-8d2b-62d85d750bb4-kube-api-access-bntjp\") pod \"26c96a23-72c8-44c6-8d2b-62d85d750bb4\" (UID: \"26c96a23-72c8-44c6-8d2b-62d85d750bb4\") "
Oct 30 18:54:40.365536 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.365508    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26c96a23-72c8-44c6-8d2b-62d85d750bb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c96a23-72c8-44c6-8d2b-62d85d750bb4-maas-api-openshift-ca\") pod \"26c96a23-72c8-44c6-8d2b-62d85d750bb4\" (UID: \"26c96a23-72c8-44c6-8d2b-62d85d750bb4\") "
Oct 30 18:54:40.365924 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.365894    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c96a23-72c8-44c6-8d2b-62d85d750bb4-maas-api-openshift-ca" pod "26c96a23-72c8-44c6-8d2b-62d85d750bb4" (UID: "26c96a23-72c8-44c6-8d2b-62d85d750bb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:40.367845 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.367796    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c96a23-72c8-44c6-8d2b-62d85d750bb4-kube-api-access-bntjp" pod "26c96a23-72c8-44c6-8d2b-62d85d750bb4" (UID: "26c96a23-72c8-44c6-8d2b-62d85d750bb4"). InnerVolumeSpecName "kube-api-access-bntjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:40.368392 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.368360    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26c96a23-72c8-44c6-8d2b-62d85d750bb4-maas-api-xks-ca" pod "26c96a23-72c8-44c6-8d2b-62d85d750bb4" (UID: "26c96a23-72c8-44c6-8d2b-62d85d750bb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:54:40.398458 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.398429    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dfda3bc-9krtl"]
Oct 30 18:54:40.467642 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.467614    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bntjp\" (UniqueName: \"kubernetes.io/projected/26c96a23-72c8-44c6-8d2b-62d85d750bb4-kube-api-access-bntjp\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:40.467812 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.467640    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c96a23-72c8-44c6-8d2b-62d85d750bb4-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:40.467812 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.467662    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c96a23-72c8-44c6-8d2b-62d85d750bb4-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:40.525700 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.525671    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e766e00b-jhpgd"]
Oct 30 18:54:40.526441 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.526417    4924 cpu_manager.go:414] "containerMap: removing container" podUID="26c96a23-72c8-44c6-8d2b-62d85d750bb4" containerName="revoke-keys"
Oct 30 18:54:40.526441 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.526439    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26c96a23-72c8-44c6-8d2b-62d85d750bb4" containerName="revoke-keys"
Oct 30 18:54:40.526570 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.526535    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="26c96a23-72c8-44c6-8d2b-62d85d750bb4" containerName="revoke-keys"
Oct 30 18:54:40.529891 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.529871    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e766e00b-jhpgd"
Oct 30 18:54:40.583620 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.583588    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e766e00b-jhpgd"]
Oct 30 18:54:40.666549 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.666368    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:40.669820 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.669783    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11a916f3-9b9b-4e2e-85d9-e19f0e14a355-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e766e00b-jhpgd\" (UID: \"11a916f3-9b9b-4e2e-85d9-e19f0e14a355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e766e00b-jhpgd"
Oct 30 18:54:40.670143 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.669898    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpmsv\" (UniqueName: \"kubernetes.io/projected/11a916f3-9b9b-4e2e-85d9-e19f0e14a355-kube-api-access-bpmsv\") pod \"maas-api-revoke-sub-e766e00b-jhpgd\" (UID: \"11a916f3-9b9b-4e2e-85d9-e19f0e14a355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e766e00b-jhpgd"
Oct 30 18:54:40.670143 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.670036    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11a916f3-9b9b-4e2e-85d9-e19f0e14a355-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e766e00b-jhpgd\" (UID: \"11a916f3-9b9b-4e2e-85d9-e19f0e14a355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e766e00b-jhpgd"
Oct 30 18:54:40.771227 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.771192    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11a916f3-9b9b-4e2e-85d9-e19f0e14a355-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e766e00b-jhpgd\" (UID: \"11a916f3-9b9b-4e2e-85d9-e19f0e14a355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e766e00b-jhpgd"
Oct 30 18:54:40.771394 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.771282    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpmsv\" (UniqueName: \"kubernetes.io/projected/11a916f3-9b9b-4e2e-85d9-e19f0e14a355-kube-api-access-bpmsv\") pod \"maas-api-revoke-sub-e766e00b-jhpgd\" (UID: \"11a916f3-9b9b-4e2e-85d9-e19f0e14a355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e766e00b-jhpgd"
Oct 30 18:54:40.771465 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.771394    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11a916f3-9b9b-4e2e-85d9-e19f0e14a355-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e766e00b-jhpgd\" (UID: \"11a916f3-9b9b-4e2e-85d9-e19f0e14a355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e766e00b-jhpgd"
Oct 30 18:54:40.772000 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.771974    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11a916f3-9b9b-4e2e-85d9-e19f0e14a355-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e766e00b-jhpgd\" (UID: \"11a916f3-9b9b-4e2e-85d9-e19f0e14a355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e766e00b-jhpgd"
Oct 30 18:54:40.773539 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.773519    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11a916f3-9b9b-4e2e-85d9-e19f0e14a355-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e766e00b-jhpgd\" (UID: \"11a916f3-9b9b-4e2e-85d9-e19f0e14a355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e766e00b-jhpgd"
Oct 30 18:54:40.825557 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.825532    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpmsv\" (UniqueName: \"kubernetes.io/projected/11a916f3-9b9b-4e2e-85d9-e19f0e14a355-kube-api-access-bpmsv\") pod \"maas-api-revoke-sub-e766e00b-jhpgd\" (UID: \"11a916f3-9b9b-4e2e-85d9-e19f0e14a355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e766e00b-jhpgd"
Oct 30 18:54:40.843221 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:40.843193    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e766e00b-jhpgd"
Oct 30 18:54:41.073093 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:41.073040    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f478c53e-wsxp2"
Oct 30 18:54:41.073093 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:41.073054    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f478c53e-wsxp2" event={"ID":"26c96a23-72c8-44c6-8d2b-62d85d750bb4","Type":"ContainerDied","Data":"2c688ef29e73a462a33baba0ad4851ae257039947e5b59a43d2995489e282f7f"}
Oct 30 18:54:41.073548 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:41.073119    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c688ef29e73a462a33baba0ad4851ae257039947e5b59a43d2995489e282f7f"
Oct 30 18:54:41.080454 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:41.080422    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dfda3bc-9krtl" event={"ID":"1a0e3b75-0597-4169-9dfa-848c4359eec3","Type":"ContainerStarted","Data":"07969b7e5541e482f86c43c7ed5f169ebe6811371301f94c608dfe19c3797315"}
Oct 30 18:54:41.080565 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:41.080514    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dfda3bc-9krtl" event={"ID":"1a0e3b75-0597-4169-9dfa-848c4359eec3","Type":"ContainerStarted","Data":"0de0546c5e6bb8ff5c4bde018705170259c04ab306a64aec62d8125d2f69a3a8"}
Oct 30 18:54:41.222825 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:54:41.222778    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a916f3_9b9b_4e2e_85d9_e19f0e14a355.slice/crio-1eefe085c472bd65d95de2a0db3917504a0a258276b4bc6fef28314054ff1b86 WatchSource:0}: Error finding container 1eefe085c472bd65d95de2a0db3917504a0a258276b4bc6fef28314054ff1b86: Status 404 returned error can't find the container with id 1eefe085c472bd65d95de2a0db3917504a0a258276b4bc6fef28314054ff1b86
Oct 30 18:54:41.240658 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:41.240634    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e766e00b-jhpgd"]
Oct 30 18:54:42.088238 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:42.088212    4924 generic.go:356] "Generic (PLEG): container finished" podID="1a0e3b75-0597-4169-9dfa-848c4359eec3" containerID="07969b7e5541e482f86c43c7ed5f169ebe6811371301f94c608dfe19c3797315" exitCode=0
Oct 30 18:54:42.088608 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:42.088297    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dfda3bc-9krtl" event={"ID":"1a0e3b75-0597-4169-9dfa-848c4359eec3","Type":"ContainerDied","Data":"07969b7e5541e482f86c43c7ed5f169ebe6811371301f94c608dfe19c3797315"}
Oct 30 18:54:42.089874 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:42.089854    4924 generic.go:356] "Generic (PLEG): container finished" podID="11a916f3-9b9b-4e2e-85d9-e19f0e14a355" containerID="451d1a8b0633a125b57a2ff6892faf63089d490b2ea05ebeff8af21e21623264" exitCode=0
Oct 30 18:54:42.089953 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:42.089902    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e766e00b-jhpgd" event={"ID":"11a916f3-9b9b-4e2e-85d9-e19f0e14a355","Type":"ContainerDied","Data":"451d1a8b0633a125b57a2ff6892faf63089d490b2ea05ebeff8af21e21623264"}
Oct 30 18:54:42.089953 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:42.089945    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e766e00b-jhpgd" event={"ID":"11a916f3-9b9b-4e2e-85d9-e19f0e14a355","Type":"ContainerStarted","Data":"1eefe085c472bd65d95de2a0db3917504a0a258276b4bc6fef28314054ff1b86"}
Oct 30 18:54:42.227372 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:42.227349    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dfda3bc-9krtl"
Oct 30 18:54:42.287347 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:42.287319    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a0e3b75-0597-4169-9dfa-848c4359eec3-kube-api-access-r2zzf\" (UniqueName: \"kubernetes.io/projected/1a0e3b75-0597-4169-9dfa-848c4359eec3-kube-api-access-r2zzf\") pod \"1a0e3b75-0597-4169-9dfa-848c4359eec3\" (UID: \"1a0e3b75-0597-4169-9dfa-848c4359eec3\") "
Oct 30 18:54:42.287503 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:42.287453    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a0e3b75-0597-4169-9dfa-848c4359eec3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a0e3b75-0597-4169-9dfa-848c4359eec3-maas-api-openshift-ca\") pod \"1a0e3b75-0597-4169-9dfa-848c4359eec3\" (UID: \"1a0e3b75-0597-4169-9dfa-848c4359eec3\") "
Oct 30 18:54:42.287503 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:42.287482    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a0e3b75-0597-4169-9dfa-848c4359eec3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a0e3b75-0597-4169-9dfa-848c4359eec3-maas-api-xks-ca\") pod \"1a0e3b75-0597-4169-9dfa-848c4359eec3\" (UID: \"1a0e3b75-0597-4169-9dfa-848c4359eec3\") "
Oct 30 18:54:42.287825 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:42.287798    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a0e3b75-0597-4169-9dfa-848c4359eec3-maas-api-openshift-ca" pod "1a0e3b75-0597-4169-9dfa-848c4359eec3" (UID: "1a0e3b75-0597-4169-9dfa-848c4359eec3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:42.288018 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:42.288000    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a0e3b75-0597-4169-9dfa-848c4359eec3-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:42.289481 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:42.289447    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a0e3b75-0597-4169-9dfa-848c4359eec3-maas-api-xks-ca" pod "1a0e3b75-0597-4169-9dfa-848c4359eec3" (UID: "1a0e3b75-0597-4169-9dfa-848c4359eec3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:54:42.289587 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:42.289568    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a0e3b75-0597-4169-9dfa-848c4359eec3-kube-api-access-r2zzf" pod "1a0e3b75-0597-4169-9dfa-848c4359eec3" (UID: "1a0e3b75-0597-4169-9dfa-848c4359eec3"). InnerVolumeSpecName "kube-api-access-r2zzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:42.389551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:42.389526    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2zzf\" (UniqueName: \"kubernetes.io/projected/1a0e3b75-0597-4169-9dfa-848c4359eec3-kube-api-access-r2zzf\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:42.389551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:42.389550    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a0e3b75-0597-4169-9dfa-848c4359eec3-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:43.096763 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:43.096731    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dfda3bc-9krtl" event={"ID":"1a0e3b75-0597-4169-9dfa-848c4359eec3","Type":"ContainerDied","Data":"0de0546c5e6bb8ff5c4bde018705170259c04ab306a64aec62d8125d2f69a3a8"}
Oct 30 18:54:43.097187 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:43.096762    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dfda3bc-9krtl"
Oct 30 18:54:43.097187 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:43.096783    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0de0546c5e6bb8ff5c4bde018705170259c04ab306a64aec62d8125d2f69a3a8"
Oct 30 18:54:43.245229 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:43.245205    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e766e00b-jhpgd"
Oct 30 18:54:43.399600 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:43.399524    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11a916f3-9b9b-4e2e-85d9-e19f0e14a355-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11a916f3-9b9b-4e2e-85d9-e19f0e14a355-maas-api-xks-ca\") pod \"11a916f3-9b9b-4e2e-85d9-e19f0e14a355\" (UID: \"11a916f3-9b9b-4e2e-85d9-e19f0e14a355\") "
Oct 30 18:54:43.399778 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:43.399617    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11a916f3-9b9b-4e2e-85d9-e19f0e14a355-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11a916f3-9b9b-4e2e-85d9-e19f0e14a355-maas-api-openshift-ca\") pod \"11a916f3-9b9b-4e2e-85d9-e19f0e14a355\" (UID: \"11a916f3-9b9b-4e2e-85d9-e19f0e14a355\") "
Oct 30 18:54:43.399778 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:43.399657    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a916f3-9b9b-4e2e-85d9-e19f0e14a355-kube-api-access-bpmsv\" (UniqueName: \"kubernetes.io/projected/11a916f3-9b9b-4e2e-85d9-e19f0e14a355-kube-api-access-bpmsv\") pod \"11a916f3-9b9b-4e2e-85d9-e19f0e14a355\" (UID: \"11a916f3-9b9b-4e2e-85d9-e19f0e14a355\") "
Oct 30 18:54:43.399980 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:43.399956    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a916f3-9b9b-4e2e-85d9-e19f0e14a355-maas-api-openshift-ca" pod "11a916f3-9b9b-4e2e-85d9-e19f0e14a355" (UID: "11a916f3-9b9b-4e2e-85d9-e19f0e14a355"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:43.400190 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:43.400170    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11a916f3-9b9b-4e2e-85d9-e19f0e14a355-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:43.401644 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:43.401620    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11a916f3-9b9b-4e2e-85d9-e19f0e14a355-maas-api-xks-ca" pod "11a916f3-9b9b-4e2e-85d9-e19f0e14a355" (UID: "11a916f3-9b9b-4e2e-85d9-e19f0e14a355"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:54:43.401730 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:43.401696    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a916f3-9b9b-4e2e-85d9-e19f0e14a355-kube-api-access-bpmsv" pod "11a916f3-9b9b-4e2e-85d9-e19f0e14a355" (UID: "11a916f3-9b9b-4e2e-85d9-e19f0e14a355"). InnerVolumeSpecName "kube-api-access-bpmsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:43.500794 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:43.500762    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11a916f3-9b9b-4e2e-85d9-e19f0e14a355-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:43.500794 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:43.500791    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpmsv\" (UniqueName: \"kubernetes.io/projected/11a916f3-9b9b-4e2e-85d9-e19f0e14a355-kube-api-access-bpmsv\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:44.104710 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:44.104683    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e766e00b-jhpgd"
Oct 30 18:54:44.104710 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:44.104701    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e766e00b-jhpgd" event={"ID":"11a916f3-9b9b-4e2e-85d9-e19f0e14a355","Type":"ContainerDied","Data":"1eefe085c472bd65d95de2a0db3917504a0a258276b4bc6fef28314054ff1b86"}
Oct 30 18:54:44.105159 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:44.104742    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1eefe085c472bd65d95de2a0db3917504a0a258276b4bc6fef28314054ff1b86"
Oct 30 18:54:45.647124 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:45.647059    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:45.711974 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:54:45.711933    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:54:50.409332 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:50.409290    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:51.405981 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:51.405949    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f752cb5-hpwb4"]
Oct 30 18:54:51.406490 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:51.406475    4924 cpu_manager.go:414] "containerMap: removing container" podUID="1a0e3b75-0597-4169-9dfa-848c4359eec3" containerName="revoke-keys"
Oct 30 18:54:51.406586 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:51.406491    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a0e3b75-0597-4169-9dfa-848c4359eec3" containerName="revoke-keys"
Oct 30 18:54:51.406586 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:51.406513    4924 cpu_manager.go:414] "containerMap: removing container" podUID="11a916f3-9b9b-4e2e-85d9-e19f0e14a355" containerName="revoke-keys"
Oct 30 18:54:51.406586 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:51.406521    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a916f3-9b9b-4e2e-85d9-e19f0e14a355" containerName="revoke-keys"
Oct 30 18:54:51.406679 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:51.406633    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a0e3b75-0597-4169-9dfa-848c4359eec3" containerName="revoke-keys"
Oct 30 18:54:51.406679 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:51.406643    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a916f3-9b9b-4e2e-85d9-e19f0e14a355" containerName="revoke-keys"
Oct 30 18:54:51.410889 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:51.410871    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f752cb5-hpwb4"
Oct 30 18:54:51.464808 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:51.464775    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f752cb5-hpwb4"]
Oct 30 18:54:51.507330 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:51.507297    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:54:51.507330 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:51.507309    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:54:51.507512 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:51.507311    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:54:51.509567 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:51.509555    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 18:54:51.585943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:51.585913    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05fc13a-ba76-459d-aa9b-cf06ba7ece60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f752cb5-hpwb4\" (UID: \"c05fc13a-ba76-459d-aa9b-cf06ba7ece60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f752cb5-hpwb4"
Oct 30 18:54:51.586144 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:51.585956    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2c4g\" (UniqueName: \"kubernetes.io/projected/c05fc13a-ba76-459d-aa9b-cf06ba7ece60-kube-api-access-s2c4g\") pod \"maas-api-revoke-sub-9f752cb5-hpwb4\" (UID: \"c05fc13a-ba76-459d-aa9b-cf06ba7ece60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f752cb5-hpwb4"
Oct 30 18:54:51.586144 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:51.586119    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05fc13a-ba76-459d-aa9b-cf06ba7ece60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f752cb5-hpwb4\" (UID: \"c05fc13a-ba76-459d-aa9b-cf06ba7ece60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f752cb5-hpwb4"
Oct 30 18:54:51.687511 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:51.687434    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05fc13a-ba76-459d-aa9b-cf06ba7ece60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f752cb5-hpwb4\" (UID: \"c05fc13a-ba76-459d-aa9b-cf06ba7ece60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f752cb5-hpwb4"
Oct 30 18:54:51.687511 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:51.687480    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2c4g\" (UniqueName: \"kubernetes.io/projected/c05fc13a-ba76-459d-aa9b-cf06ba7ece60-kube-api-access-s2c4g\") pod \"maas-api-revoke-sub-9f752cb5-hpwb4\" (UID: \"c05fc13a-ba76-459d-aa9b-cf06ba7ece60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f752cb5-hpwb4"
Oct 30 18:54:51.687736 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:51.687604    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05fc13a-ba76-459d-aa9b-cf06ba7ece60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f752cb5-hpwb4\" (UID: \"c05fc13a-ba76-459d-aa9b-cf06ba7ece60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f752cb5-hpwb4"
Oct 30 18:54:51.688115 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:51.688093    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05fc13a-ba76-459d-aa9b-cf06ba7ece60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f752cb5-hpwb4\" (UID: \"c05fc13a-ba76-459d-aa9b-cf06ba7ece60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f752cb5-hpwb4"
Oct 30 18:54:51.689881 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:51.689861    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05fc13a-ba76-459d-aa9b-cf06ba7ece60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f752cb5-hpwb4\" (UID: \"c05fc13a-ba76-459d-aa9b-cf06ba7ece60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f752cb5-hpwb4"
Oct 30 18:54:51.742796 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:51.742769    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2c4g\" (UniqueName: \"kubernetes.io/projected/c05fc13a-ba76-459d-aa9b-cf06ba7ece60-kube-api-access-s2c4g\") pod \"maas-api-revoke-sub-9f752cb5-hpwb4\" (UID: \"c05fc13a-ba76-459d-aa9b-cf06ba7ece60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f752cb5-hpwb4"
Oct 30 18:54:52.024968 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:52.024890    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f752cb5-hpwb4"
Oct 30 18:54:52.391884 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:52.391862    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f752cb5-hpwb4"]
Oct 30 18:54:52.393330 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:54:52.393294    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc05fc13a_ba76_459d_aa9b_cf06ba7ece60.slice/crio-ca8f5d119021930419dfb2d5f6a7103bdd0a24d2e26f25cb78fc0981ecd65119 WatchSource:0}: Error finding container ca8f5d119021930419dfb2d5f6a7103bdd0a24d2e26f25cb78fc0981ecd65119: Status 404 returned error can't find the container with id ca8f5d119021930419dfb2d5f6a7103bdd0a24d2e26f25cb78fc0981ecd65119
Oct 30 18:54:53.160814 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:53.160780    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f752cb5-hpwb4" event={"ID":"c05fc13a-ba76-459d-aa9b-cf06ba7ece60","Type":"ContainerStarted","Data":"a9cfae4340d8a4d1ab301b8d6a3fde9f38fa12e305f944ca27af5901ccde4bae"}
Oct 30 18:54:53.161317 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:53.160832    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f752cb5-hpwb4" event={"ID":"c05fc13a-ba76-459d-aa9b-cf06ba7ece60","Type":"ContainerStarted","Data":"ca8f5d119021930419dfb2d5f6a7103bdd0a24d2e26f25cb78fc0981ecd65119"}
Oct 30 18:54:53.275320 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:53.275264    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f752cb5-hpwb4" podStartSLOduration=2.275245099 podStartE2EDuration="2.275245099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:53.269997428 +0000 UTC m=+1094.547047874" watchObservedRunningTime="2026-10-30 18:54:53.275245099 +0000 UTC m=+1094.552295540"
Oct 30 18:54:54.169284 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:54.169248    4924 generic.go:356] "Generic (PLEG): container finished" podID="c05fc13a-ba76-459d-aa9b-cf06ba7ece60" containerID="a9cfae4340d8a4d1ab301b8d6a3fde9f38fa12e305f944ca27af5901ccde4bae" exitCode=0
Oct 30 18:54:54.169789 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:54.169365    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f752cb5-hpwb4" event={"ID":"c05fc13a-ba76-459d-aa9b-cf06ba7ece60","Type":"ContainerDied","Data":"a9cfae4340d8a4d1ab301b8d6a3fde9f38fa12e305f944ca27af5901ccde4bae"}
Oct 30 18:54:54.336088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:54.335809    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2adf84ef-vqcrt"]
Oct 30 18:54:54.369595 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:54.369562    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2adf84ef-vqcrt"
Oct 30 18:54:54.403267 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:54.403232    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2adf84ef-vqcrt"]
Oct 30 18:54:54.554998 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:54.554896    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tc2m\" (UniqueName: \"kubernetes.io/projected/eb9ecacf-1867-4a7f-af5b-1b0ab354a66b-kube-api-access-5tc2m\") pod \"maas-api-revoke-sub-2adf84ef-vqcrt\" (UID: \"eb9ecacf-1867-4a7f-af5b-1b0ab354a66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2adf84ef-vqcrt"
Oct 30 18:54:54.555330 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:54.555302    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb9ecacf-1867-4a7f-af5b-1b0ab354a66b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2adf84ef-vqcrt\" (UID: \"eb9ecacf-1867-4a7f-af5b-1b0ab354a66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2adf84ef-vqcrt"
Oct 30 18:54:54.555655 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:54.555634    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb9ecacf-1867-4a7f-af5b-1b0ab354a66b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2adf84ef-vqcrt\" (UID: \"eb9ecacf-1867-4a7f-af5b-1b0ab354a66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2adf84ef-vqcrt"
Oct 30 18:54:54.658109 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:54.657991    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tc2m\" (UniqueName: \"kubernetes.io/projected/eb9ecacf-1867-4a7f-af5b-1b0ab354a66b-kube-api-access-5tc2m\") pod \"maas-api-revoke-sub-2adf84ef-vqcrt\" (UID: \"eb9ecacf-1867-4a7f-af5b-1b0ab354a66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2adf84ef-vqcrt"
Oct 30 18:54:54.665132 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:54.658793    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb9ecacf-1867-4a7f-af5b-1b0ab354a66b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2adf84ef-vqcrt\" (UID: \"eb9ecacf-1867-4a7f-af5b-1b0ab354a66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2adf84ef-vqcrt"
Oct 30 18:54:54.665132 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:54.658970    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb9ecacf-1867-4a7f-af5b-1b0ab354a66b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2adf84ef-vqcrt\" (UID: \"eb9ecacf-1867-4a7f-af5b-1b0ab354a66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2adf84ef-vqcrt"
Oct 30 18:54:54.665132 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:54.659743    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb9ecacf-1867-4a7f-af5b-1b0ab354a66b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2adf84ef-vqcrt\" (UID: \"eb9ecacf-1867-4a7f-af5b-1b0ab354a66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2adf84ef-vqcrt"
Oct 30 18:54:54.697091 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:54.694169    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb9ecacf-1867-4a7f-af5b-1b0ab354a66b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2adf84ef-vqcrt\" (UID: \"eb9ecacf-1867-4a7f-af5b-1b0ab354a66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2adf84ef-vqcrt"
Oct 30 18:54:54.734089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:54.727019    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tc2m\" (UniqueName: \"kubernetes.io/projected/eb9ecacf-1867-4a7f-af5b-1b0ab354a66b-kube-api-access-5tc2m\") pod \"maas-api-revoke-sub-2adf84ef-vqcrt\" (UID: \"eb9ecacf-1867-4a7f-af5b-1b0ab354a66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2adf84ef-vqcrt"
Oct 30 18:54:55.022089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:55.008980    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2adf84ef-vqcrt"
Oct 30 18:54:55.499023 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:55.498998    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f752cb5-hpwb4"
Oct 30 18:54:55.594934 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:55.594909    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c05fc13a-ba76-459d-aa9b-cf06ba7ece60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05fc13a-ba76-459d-aa9b-cf06ba7ece60-maas-api-openshift-ca\") pod \"c05fc13a-ba76-459d-aa9b-cf06ba7ece60\" (UID: \"c05fc13a-ba76-459d-aa9b-cf06ba7ece60\") "
Oct 30 18:54:55.595076 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:55.595049    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c05fc13a-ba76-459d-aa9b-cf06ba7ece60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05fc13a-ba76-459d-aa9b-cf06ba7ece60-maas-api-xks-ca\") pod \"c05fc13a-ba76-459d-aa9b-cf06ba7ece60\" (UID: \"c05fc13a-ba76-459d-aa9b-cf06ba7ece60\") "
Oct 30 18:54:55.595134 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:55.595110    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c05fc13a-ba76-459d-aa9b-cf06ba7ece60-kube-api-access-s2c4g\" (UniqueName: \"kubernetes.io/projected/c05fc13a-ba76-459d-aa9b-cf06ba7ece60-kube-api-access-s2c4g\") pod \"c05fc13a-ba76-459d-aa9b-cf06ba7ece60\" (UID: \"c05fc13a-ba76-459d-aa9b-cf06ba7ece60\") "
Oct 30 18:54:55.595327 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:55.595300    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c05fc13a-ba76-459d-aa9b-cf06ba7ece60-maas-api-openshift-ca" pod "c05fc13a-ba76-459d-aa9b-cf06ba7ece60" (UID: "c05fc13a-ba76-459d-aa9b-cf06ba7ece60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:55.595715 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:55.595693    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c05fc13a-ba76-459d-aa9b-cf06ba7ece60-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:55.596898 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:54:55.596871    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9ecacf_1867_4a7f_af5b_1b0ab354a66b.slice/crio-d588a9ddafbdfb9f613abf73084330f9af1e5dc79d01b05379bb54a36abcc560 WatchSource:0}: Error finding container d588a9ddafbdfb9f613abf73084330f9af1e5dc79d01b05379bb54a36abcc560: Status 404 returned error can't find the container with id d588a9ddafbdfb9f613abf73084330f9af1e5dc79d01b05379bb54a36abcc560
Oct 30 18:54:55.597515 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:55.597488    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c05fc13a-ba76-459d-aa9b-cf06ba7ece60-maas-api-xks-ca" pod "c05fc13a-ba76-459d-aa9b-cf06ba7ece60" (UID: "c05fc13a-ba76-459d-aa9b-cf06ba7ece60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:54:55.597657 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:55.597642    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c05fc13a-ba76-459d-aa9b-cf06ba7ece60-kube-api-access-s2c4g" pod "c05fc13a-ba76-459d-aa9b-cf06ba7ece60" (UID: "c05fc13a-ba76-459d-aa9b-cf06ba7ece60"). InnerVolumeSpecName "kube-api-access-s2c4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:55.599211 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:55.599189    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2adf84ef-vqcrt"]
Oct 30 18:54:55.696776 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:55.696745    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c05fc13a-ba76-459d-aa9b-cf06ba7ece60-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:55.696776 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:55.696773    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2c4g\" (UniqueName: \"kubernetes.io/projected/c05fc13a-ba76-459d-aa9b-cf06ba7ece60-kube-api-access-s2c4g\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:56.029644 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:56.029614    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:56.204202 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:56.204107    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2adf84ef-vqcrt" event={"ID":"eb9ecacf-1867-4a7f-af5b-1b0ab354a66b","Type":"ContainerStarted","Data":"480ed17a992e5b1e18ad09d44c2a0deee5d5f0be2d55021ae0c122dd4bb270e0"}
Oct 30 18:54:56.204202 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:56.204167    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2adf84ef-vqcrt" event={"ID":"eb9ecacf-1867-4a7f-af5b-1b0ab354a66b","Type":"ContainerStarted","Data":"d588a9ddafbdfb9f613abf73084330f9af1e5dc79d01b05379bb54a36abcc560"}
Oct 30 18:54:56.205620 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:56.205599    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f752cb5-hpwb4"
Oct 30 18:54:56.205753 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:56.205646    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f752cb5-hpwb4" event={"ID":"c05fc13a-ba76-459d-aa9b-cf06ba7ece60","Type":"ContainerDied","Data":"ca8f5d119021930419dfb2d5f6a7103bdd0a24d2e26f25cb78fc0981ecd65119"}
Oct 30 18:54:56.205753 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:56.205695    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca8f5d119021930419dfb2d5f6a7103bdd0a24d2e26f25cb78fc0981ecd65119"
Oct 30 18:54:56.318766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:56.318717    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2adf84ef-vqcrt" podStartSLOduration=2.318703588 podStartE2EDuration="2.318703588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:56.31542995 +0000 UTC m=+1097.592480394" watchObservedRunningTime="2026-10-30 18:54:56.318703588 +0000 UTC m=+1097.595754070"
Oct 30 18:54:57.213440 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.213408    4924 generic.go:356] "Generic (PLEG): container finished" podID="eb9ecacf-1867-4a7f-af5b-1b0ab354a66b" containerID="480ed17a992e5b1e18ad09d44c2a0deee5d5f0be2d55021ae0c122dd4bb270e0" exitCode=0
Oct 30 18:54:57.213851 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.213498    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2adf84ef-vqcrt" event={"ID":"eb9ecacf-1867-4a7f-af5b-1b0ab354a66b","Type":"ContainerDied","Data":"480ed17a992e5b1e18ad09d44c2a0deee5d5f0be2d55021ae0c122dd4bb270e0"}
Oct 30 18:54:57.371373 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.371342    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21eb612d-jjp9d"]
Oct 30 18:54:57.371920 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.371904    4924 cpu_manager.go:414] "containerMap: removing container" podUID="c05fc13a-ba76-459d-aa9b-cf06ba7ece60" containerName="revoke-keys"
Oct 30 18:54:57.371986 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.371921    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c05fc13a-ba76-459d-aa9b-cf06ba7ece60" containerName="revoke-keys"
Oct 30 18:54:57.372054 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.372043    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="c05fc13a-ba76-459d-aa9b-cf06ba7ece60" containerName="revoke-keys"
Oct 30 18:54:57.385330 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.385298    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21eb612d-jjp9d"
Oct 30 18:54:57.387738 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.387714    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21eb612d-jjp9d"]
Oct 30 18:54:57.472558 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.472485    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"]
Oct 30 18:54:57.476548 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.476528    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:54:57.516658 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.516624    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9rtx\" (UniqueName: \"kubernetes.io/projected/7eb70edf-668c-4255-a633-e52ea0f587da-kube-api-access-d9rtx\") pod \"maas-api-revoke-sub-21eb612d-jjp9d\" (UID: \"7eb70edf-668c-4255-a633-e52ea0f587da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21eb612d-jjp9d"
Oct 30 18:54:57.516798 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.516695    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb70edf-668c-4255-a633-e52ea0f587da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21eb612d-jjp9d\" (UID: \"7eb70edf-668c-4255-a633-e52ea0f587da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21eb612d-jjp9d"
Oct 30 18:54:57.516841 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.516820    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb70edf-668c-4255-a633-e52ea0f587da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21eb612d-jjp9d\" (UID: \"7eb70edf-668c-4255-a633-e52ea0f587da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21eb612d-jjp9d"
Oct 30 18:54:57.526896 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.526861    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ec8dab\"/\"distinct-w2-ec8dab-kserve-self-signed-certs\""
Oct 30 18:54:57.543128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.543097    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"]
Oct 30 18:54:57.617845 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.617806    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb70edf-668c-4255-a633-e52ea0f587da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21eb612d-jjp9d\" (UID: \"7eb70edf-668c-4255-a633-e52ea0f587da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21eb612d-jjp9d"
Oct 30 18:54:57.618036 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.617884    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-home\") pod \"distinct-w2-ec8dab-kserve-76788c545b-26h86\" (UID: \"60a58597-fdc4-4722-8d2a-29e475f84a76\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:54:57.618036 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.617910    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s4bl\" (UniqueName: \"kubernetes.io/projected/60a58597-fdc4-4722-8d2a-29e475f84a76-kube-api-access-5s4bl\") pod \"distinct-w2-ec8dab-kserve-76788c545b-26h86\" (UID: \"60a58597-fdc4-4722-8d2a-29e475f84a76\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:54:57.618194 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.618042    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-dshm\") pod \"distinct-w2-ec8dab-kserve-76788c545b-26h86\" (UID: \"60a58597-fdc4-4722-8d2a-29e475f84a76\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:54:57.618194 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.618130    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-model-cache\") pod \"distinct-w2-ec8dab-kserve-76788c545b-26h86\" (UID: \"60a58597-fdc4-4722-8d2a-29e475f84a76\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:54:57.618194 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.618184    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60a58597-fdc4-4722-8d2a-29e475f84a76-tls-certs\") pod \"distinct-w2-ec8dab-kserve-76788c545b-26h86\" (UID: \"60a58597-fdc4-4722-8d2a-29e475f84a76\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:54:57.618334 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.618310    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-tmp-dir\") pod \"distinct-w2-ec8dab-kserve-76788c545b-26h86\" (UID: \"60a58597-fdc4-4722-8d2a-29e475f84a76\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:54:57.618377 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.618343    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9rtx\" (UniqueName: \"kubernetes.io/projected/7eb70edf-668c-4255-a633-e52ea0f587da-kube-api-access-d9rtx\") pod \"maas-api-revoke-sub-21eb612d-jjp9d\" (UID: \"7eb70edf-668c-4255-a633-e52ea0f587da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21eb612d-jjp9d"
Oct 30 18:54:57.618377 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.618374    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb70edf-668c-4255-a633-e52ea0f587da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21eb612d-jjp9d\" (UID: \"7eb70edf-668c-4255-a633-e52ea0f587da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21eb612d-jjp9d"
Oct 30 18:54:57.618889 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.618871    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb70edf-668c-4255-a633-e52ea0f587da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21eb612d-jjp9d\" (UID: \"7eb70edf-668c-4255-a633-e52ea0f587da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21eb612d-jjp9d"
Oct 30 18:54:57.620192 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.620173    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb70edf-668c-4255-a633-e52ea0f587da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21eb612d-jjp9d\" (UID: \"7eb70edf-668c-4255-a633-e52ea0f587da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21eb612d-jjp9d"
Oct 30 18:54:57.625616 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.625592    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:54:57.675567 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.674942    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9rtx\" (UniqueName: \"kubernetes.io/projected/7eb70edf-668c-4255-a633-e52ea0f587da-kube-api-access-d9rtx\") pod \"maas-api-revoke-sub-21eb612d-jjp9d\" (UID: \"7eb70edf-668c-4255-a633-e52ea0f587da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21eb612d-jjp9d"
Oct 30 18:54:57.700905 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.700852    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21eb612d-jjp9d"
Oct 30 18:54:57.719776 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.719740    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-home\") pod \"distinct-w2-ec8dab-kserve-76788c545b-26h86\" (UID: \"60a58597-fdc4-4722-8d2a-29e475f84a76\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:54:57.719904 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.719794    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s4bl\" (UniqueName: \"kubernetes.io/projected/60a58597-fdc4-4722-8d2a-29e475f84a76-kube-api-access-5s4bl\") pod \"distinct-w2-ec8dab-kserve-76788c545b-26h86\" (UID: \"60a58597-fdc4-4722-8d2a-29e475f84a76\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:54:57.719904 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.719862    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-dshm\") pod \"distinct-w2-ec8dab-kserve-76788c545b-26h86\" (UID: \"60a58597-fdc4-4722-8d2a-29e475f84a76\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:54:57.719904 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.719892    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-model-cache\") pod \"distinct-w2-ec8dab-kserve-76788c545b-26h86\" (UID: \"60a58597-fdc4-4722-8d2a-29e475f84a76\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:54:57.720055 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.719939    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60a58597-fdc4-4722-8d2a-29e475f84a76-tls-certs\") pod \"distinct-w2-ec8dab-kserve-76788c545b-26h86\" (UID: \"60a58597-fdc4-4722-8d2a-29e475f84a76\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:54:57.720055 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.720022    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-tmp-dir\") pod \"distinct-w2-ec8dab-kserve-76788c545b-26h86\" (UID: \"60a58597-fdc4-4722-8d2a-29e475f84a76\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:54:57.720211 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.720189    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-home\") pod \"distinct-w2-ec8dab-kserve-76788c545b-26h86\" (UID: \"60a58597-fdc4-4722-8d2a-29e475f84a76\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:54:57.720372 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.720348    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-model-cache\") pod \"distinct-w2-ec8dab-kserve-76788c545b-26h86\" (UID: \"60a58597-fdc4-4722-8d2a-29e475f84a76\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:54:57.720456 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.720377    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-tmp-dir\") pod \"distinct-w2-ec8dab-kserve-76788c545b-26h86\" (UID: \"60a58597-fdc4-4722-8d2a-29e475f84a76\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:54:57.722322 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.722298    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-dshm\") pod \"distinct-w2-ec8dab-kserve-76788c545b-26h86\" (UID: \"60a58597-fdc4-4722-8d2a-29e475f84a76\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:54:57.722752 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.722706    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60a58597-fdc4-4722-8d2a-29e475f84a76-tls-certs\") pod \"distinct-w2-ec8dab-kserve-76788c545b-26h86\" (UID: \"60a58597-fdc4-4722-8d2a-29e475f84a76\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:54:57.780099 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.776043    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s4bl\" (UniqueName: \"kubernetes.io/projected/60a58597-fdc4-4722-8d2a-29e475f84a76-kube-api-access-5s4bl\") pod \"distinct-w2-ec8dab-kserve-76788c545b-26h86\" (UID: \"60a58597-fdc4-4722-8d2a-29e475f84a76\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:54:57.788033 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:57.788012    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:54:58.070351 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:58.070323    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21eb612d-jjp9d"]
Oct 30 18:54:58.070527 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:54:58.070507    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb70edf_668c_4255_a633_e52ea0f587da.slice/crio-3fd6058651a7713630efae529a4ac0d28db0c10163e270e29543b7409d973be2 WatchSource:0}: Error finding container 3fd6058651a7713630efae529a4ac0d28db0c10163e270e29543b7409d973be2: Status 404 returned error can't find the container with id 3fd6058651a7713630efae529a4ac0d28db0c10163e270e29543b7409d973be2
Oct 30 18:54:58.165850 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:58.165817    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"]
Oct 30 18:54:58.166391 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:54:58.166363    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60a58597_fdc4_4722_8d2a_29e475f84a76.slice/crio-bc886d1c332bd5311ee94af204769eb360f84a0dda4db7ca0214fc886c528c06 WatchSource:0}: Error finding container bc886d1c332bd5311ee94af204769eb360f84a0dda4db7ca0214fc886c528c06: Status 404 returned error can't find the container with id bc886d1c332bd5311ee94af204769eb360f84a0dda4db7ca0214fc886c528c06
Oct 30 18:54:58.221535 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:58.221499    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86" event={"ID":"60a58597-fdc4-4722-8d2a-29e475f84a76","Type":"ContainerStarted","Data":"bc886d1c332bd5311ee94af204769eb360f84a0dda4db7ca0214fc886c528c06"}
Oct 30 18:54:58.223017 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:58.222990    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21eb612d-jjp9d" event={"ID":"7eb70edf-668c-4255-a633-e52ea0f587da","Type":"ContainerStarted","Data":"3fd6058651a7713630efae529a4ac0d28db0c10163e270e29543b7409d973be2"}
Oct 30 18:54:58.361001 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:58.360976    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2adf84ef-vqcrt"
Oct 30 18:54:58.530482 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:58.530444    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb9ecacf-1867-4a7f-af5b-1b0ab354a66b-kube-api-access-5tc2m\" (UniqueName: \"kubernetes.io/projected/eb9ecacf-1867-4a7f-af5b-1b0ab354a66b-kube-api-access-5tc2m\") pod \"eb9ecacf-1867-4a7f-af5b-1b0ab354a66b\" (UID: \"eb9ecacf-1867-4a7f-af5b-1b0ab354a66b\") "
Oct 30 18:54:58.530482 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:58.530481    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb9ecacf-1867-4a7f-af5b-1b0ab354a66b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb9ecacf-1867-4a7f-af5b-1b0ab354a66b-maas-api-xks-ca\") pod \"eb9ecacf-1867-4a7f-af5b-1b0ab354a66b\" (UID: \"eb9ecacf-1867-4a7f-af5b-1b0ab354a66b\") "
Oct 30 18:54:58.530676 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:58.530529    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb9ecacf-1867-4a7f-af5b-1b0ab354a66b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb9ecacf-1867-4a7f-af5b-1b0ab354a66b-maas-api-openshift-ca\") pod \"eb9ecacf-1867-4a7f-af5b-1b0ab354a66b\" (UID: \"eb9ecacf-1867-4a7f-af5b-1b0ab354a66b\") "
Oct 30 18:54:58.530953 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:58.530930    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb9ecacf-1867-4a7f-af5b-1b0ab354a66b-maas-api-openshift-ca" pod "eb9ecacf-1867-4a7f-af5b-1b0ab354a66b" (UID: "eb9ecacf-1867-4a7f-af5b-1b0ab354a66b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:54:58.532619 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:58.532596    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb9ecacf-1867-4a7f-af5b-1b0ab354a66b-kube-api-access-5tc2m" pod "eb9ecacf-1867-4a7f-af5b-1b0ab354a66b" (UID: "eb9ecacf-1867-4a7f-af5b-1b0ab354a66b"). InnerVolumeSpecName "kube-api-access-5tc2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:54:58.532729 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:58.532667    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb9ecacf-1867-4a7f-af5b-1b0ab354a66b-maas-api-xks-ca" pod "eb9ecacf-1867-4a7f-af5b-1b0ab354a66b" (UID: "eb9ecacf-1867-4a7f-af5b-1b0ab354a66b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:54:58.632740 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:58.632652    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tc2m\" (UniqueName: \"kubernetes.io/projected/eb9ecacf-1867-4a7f-af5b-1b0ab354a66b-kube-api-access-5tc2m\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:58.632740 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:58.632693    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb9ecacf-1867-4a7f-af5b-1b0ab354a66b-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:58.632740 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:58.632708    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb9ecacf-1867-4a7f-af5b-1b0ab354a66b-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:54:59.231380 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:59.231298    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86" event={"ID":"60a58597-fdc4-4722-8d2a-29e475f84a76","Type":"ContainerStarted","Data":"480ca2e8a1a174a796513c83a2316f1e00d053c13352d0c7790b03d7aabd619a"}
Oct 30 18:54:59.232637 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:59.232609    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2adf84ef-vqcrt"
Oct 30 18:54:59.232763 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:59.232608    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2adf84ef-vqcrt" event={"ID":"eb9ecacf-1867-4a7f-af5b-1b0ab354a66b","Type":"ContainerDied","Data":"d588a9ddafbdfb9f613abf73084330f9af1e5dc79d01b05379bb54a36abcc560"}
Oct 30 18:54:59.232763 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:59.232726    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d588a9ddafbdfb9f613abf73084330f9af1e5dc79d01b05379bb54a36abcc560"
Oct 30 18:54:59.234121 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:59.234102    4924 generic.go:356] "Generic (PLEG): container finished" podID="7eb70edf-668c-4255-a633-e52ea0f587da" containerID="b900612bb5cfa9e8958e93d14adc5e1657f774872148ea9c719d652c73bdc70c" exitCode=0
Oct 30 18:54:59.234189 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:59.234172    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21eb612d-jjp9d" event={"ID":"7eb70edf-668c-4255-a633-e52ea0f587da","Type":"ContainerDied","Data":"b900612bb5cfa9e8958e93d14adc5e1657f774872148ea9c719d652c73bdc70c"}
Oct 30 18:54:59.341023 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:59.340970    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86" podStartSLOduration=2.340955602 podStartE2EDuration="2.340955602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:59.339702262 +0000 UTC m=+1100.616752709" watchObservedRunningTime="2026-10-30 18:54:59.340955602 +0000 UTC m=+1100.618006077"
Oct 30 18:54:59.788767 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:54:59.788731    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:55:00.404567 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:00.404544    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21eb612d-jjp9d"
Oct 30 18:55:00.454423 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:00.454394    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eb70edf-668c-4255-a633-e52ea0f587da-kube-api-access-d9rtx\" (UniqueName: \"kubernetes.io/projected/7eb70edf-668c-4255-a633-e52ea0f587da-kube-api-access-d9rtx\") pod \"7eb70edf-668c-4255-a633-e52ea0f587da\" (UID: \"7eb70edf-668c-4255-a633-e52ea0f587da\") "
Oct 30 18:55:00.454575 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:00.454468    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eb70edf-668c-4255-a633-e52ea0f587da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb70edf-668c-4255-a633-e52ea0f587da-maas-api-openshift-ca\") pod \"7eb70edf-668c-4255-a633-e52ea0f587da\" (UID: \"7eb70edf-668c-4255-a633-e52ea0f587da\") "
Oct 30 18:55:00.454575 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:00.454513    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eb70edf-668c-4255-a633-e52ea0f587da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb70edf-668c-4255-a633-e52ea0f587da-maas-api-xks-ca\") pod \"7eb70edf-668c-4255-a633-e52ea0f587da\" (UID: \"7eb70edf-668c-4255-a633-e52ea0f587da\") "
Oct 30 18:55:00.454840 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:00.454816    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eb70edf-668c-4255-a633-e52ea0f587da-maas-api-openshift-ca" pod "7eb70edf-668c-4255-a633-e52ea0f587da" (UID: "7eb70edf-668c-4255-a633-e52ea0f587da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:00.456382 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:00.456363    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb70edf-668c-4255-a633-e52ea0f587da-kube-api-access-d9rtx" pod "7eb70edf-668c-4255-a633-e52ea0f587da" (UID: "7eb70edf-668c-4255-a633-e52ea0f587da"). InnerVolumeSpecName "kube-api-access-d9rtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:00.456524 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:00.456510    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eb70edf-668c-4255-a633-e52ea0f587da-maas-api-xks-ca" pod "7eb70edf-668c-4255-a633-e52ea0f587da" (UID: "7eb70edf-668c-4255-a633-e52ea0f587da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:00.556253 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:00.556169    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb70edf-668c-4255-a633-e52ea0f587da-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:00.556253 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:00.556206    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb70edf-668c-4255-a633-e52ea0f587da-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:00.556253 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:00.556216    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9rtx\" (UniqueName: \"kubernetes.io/projected/7eb70edf-668c-4255-a633-e52ea0f587da-kube-api-access-d9rtx\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:00.857564 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:00.857352    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:55:00.924114 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:00.924060    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"]
Oct 30 18:55:00.924301 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:00.924281    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn" podUID="1e919a72-c6c1-4c1f-9e24-f7c5910021de" containerName="istio-proxy" containerID="cri-o://361e8e91eb1c6cbbde4411a2bb7a06085f708e45d17b5facf349fa8c5b6e250e" gracePeriod=30
Oct 30 18:55:01.248418 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:01.248340    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21eb612d-jjp9d"
Oct 30 18:55:01.248418 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:01.248372    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21eb612d-jjp9d" event={"ID":"7eb70edf-668c-4255-a633-e52ea0f587da","Type":"ContainerDied","Data":"3fd6058651a7713630efae529a4ac0d28db0c10163e270e29543b7409d973be2"}
Oct 30 18:55:01.248418 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:01.248421    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fd6058651a7713630efae529a4ac0d28db0c10163e270e29543b7409d973be2"
Oct 30 18:55:05.012109 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.012058    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a7b9ed6-8tvfd"]
Oct 30 18:55:05.012803 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.012780    4924 cpu_manager.go:414] "containerMap: removing container" podUID="eb9ecacf-1867-4a7f-af5b-1b0ab354a66b" containerName="revoke-keys"
Oct 30 18:55:05.012965 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.012804    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb9ecacf-1867-4a7f-af5b-1b0ab354a66b" containerName="revoke-keys"
Oct 30 18:55:05.012965 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.012836    4924 cpu_manager.go:414] "containerMap: removing container" podUID="7eb70edf-668c-4255-a633-e52ea0f587da" containerName="revoke-keys"
Oct 30 18:55:05.012965 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.012844    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eb70edf-668c-4255-a633-e52ea0f587da" containerName="revoke-keys"
Oct 30 18:55:05.013143 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.012985    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb9ecacf-1867-4a7f-af5b-1b0ab354a66b" containerName="revoke-keys"
Oct 30 18:55:05.013143 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.013003    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eb70edf-668c-4255-a633-e52ea0f587da" containerName="revoke-keys"
Oct 30 18:55:05.016676 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.016657    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7b9ed6-8tvfd"
Oct 30 18:55:05.110700 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.110666    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:55:05.111402 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.111376    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca24639a-0332-4c5d-94a8-c7d8b15c6602-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a7b9ed6-8tvfd\" (UID: \"ca24639a-0332-4c5d-94a8-c7d8b15c6602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7b9ed6-8tvfd"
Oct 30 18:55:05.111488 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.111414    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbqgc\" (UniqueName: \"kubernetes.io/projected/ca24639a-0332-4c5d-94a8-c7d8b15c6602-kube-api-access-lbqgc\") pod \"maas-api-revoke-sub-6a7b9ed6-8tvfd\" (UID: \"ca24639a-0332-4c5d-94a8-c7d8b15c6602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7b9ed6-8tvfd"
Oct 30 18:55:05.111602 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.111581    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca24639a-0332-4c5d-94a8-c7d8b15c6602-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a7b9ed6-8tvfd\" (UID: \"ca24639a-0332-4c5d-94a8-c7d8b15c6602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7b9ed6-8tvfd"
Oct 30 18:55:05.112953 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.112928    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:55:05.113104 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.113040    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:55:05.113104 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.113040    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 18:55:05.114083 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.114043    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a7b9ed6-8tvfd"]
Oct 30 18:55:05.213027 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.212995    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca24639a-0332-4c5d-94a8-c7d8b15c6602-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a7b9ed6-8tvfd\" (UID: \"ca24639a-0332-4c5d-94a8-c7d8b15c6602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7b9ed6-8tvfd"
Oct 30 18:55:05.213027 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.213031    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbqgc\" (UniqueName: \"kubernetes.io/projected/ca24639a-0332-4c5d-94a8-c7d8b15c6602-kube-api-access-lbqgc\") pod \"maas-api-revoke-sub-6a7b9ed6-8tvfd\" (UID: \"ca24639a-0332-4c5d-94a8-c7d8b15c6602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7b9ed6-8tvfd"
Oct 30 18:55:05.213234 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.213109    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca24639a-0332-4c5d-94a8-c7d8b15c6602-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a7b9ed6-8tvfd\" (UID: \"ca24639a-0332-4c5d-94a8-c7d8b15c6602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7b9ed6-8tvfd"
Oct 30 18:55:05.213661 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.213629    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca24639a-0332-4c5d-94a8-c7d8b15c6602-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a7b9ed6-8tvfd\" (UID: \"ca24639a-0332-4c5d-94a8-c7d8b15c6602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7b9ed6-8tvfd"
Oct 30 18:55:05.215433 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.215411    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca24639a-0332-4c5d-94a8-c7d8b15c6602-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a7b9ed6-8tvfd\" (UID: \"ca24639a-0332-4c5d-94a8-c7d8b15c6602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7b9ed6-8tvfd"
Oct 30 18:55:05.268023 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.267965    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbqgc\" (UniqueName: \"kubernetes.io/projected/ca24639a-0332-4c5d-94a8-c7d8b15c6602-kube-api-access-lbqgc\") pod \"maas-api-revoke-sub-6a7b9ed6-8tvfd\" (UID: \"ca24639a-0332-4c5d-94a8-c7d8b15c6602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7b9ed6-8tvfd"
Oct 30 18:55:05.330731 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.330705    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7b9ed6-8tvfd"
Oct 30 18:55:05.351093 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.351041    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac2a0f50-xw5xh"]
Oct 30 18:55:05.356272 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.356250    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2a0f50-xw5xh"
Oct 30 18:55:05.414884 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.411875    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac2a0f50-xw5xh"]
Oct 30 18:55:05.415056 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.414994    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1e6fa3-8aac-4733-8e3a-a1417428e1cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac2a0f50-xw5xh\" (UID: \"2a1e6fa3-8aac-4733-8e3a-a1417428e1cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2a0f50-xw5xh"
Oct 30 18:55:05.415144 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.415107    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk9gg\" (UniqueName: \"kubernetes.io/projected/2a1e6fa3-8aac-4733-8e3a-a1417428e1cb-kube-api-access-fk9gg\") pod \"maas-api-revoke-sub-ac2a0f50-xw5xh\" (UID: \"2a1e6fa3-8aac-4733-8e3a-a1417428e1cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2a0f50-xw5xh"
Oct 30 18:55:05.415201 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.415158    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1e6fa3-8aac-4733-8e3a-a1417428e1cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac2a0f50-xw5xh\" (UID: \"2a1e6fa3-8aac-4733-8e3a-a1417428e1cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2a0f50-xw5xh"
Oct 30 18:55:05.516840 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.516805    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1e6fa3-8aac-4733-8e3a-a1417428e1cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac2a0f50-xw5xh\" (UID: \"2a1e6fa3-8aac-4733-8e3a-a1417428e1cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2a0f50-xw5xh"
Oct 30 18:55:05.517021 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.516868    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk9gg\" (UniqueName: \"kubernetes.io/projected/2a1e6fa3-8aac-4733-8e3a-a1417428e1cb-kube-api-access-fk9gg\") pod \"maas-api-revoke-sub-ac2a0f50-xw5xh\" (UID: \"2a1e6fa3-8aac-4733-8e3a-a1417428e1cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2a0f50-xw5xh"
Oct 30 18:55:05.517021 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.516899    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1e6fa3-8aac-4733-8e3a-a1417428e1cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac2a0f50-xw5xh\" (UID: \"2a1e6fa3-8aac-4733-8e3a-a1417428e1cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2a0f50-xw5xh"
Oct 30 18:55:05.517629 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.517608    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1e6fa3-8aac-4733-8e3a-a1417428e1cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac2a0f50-xw5xh\" (UID: \"2a1e6fa3-8aac-4733-8e3a-a1417428e1cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2a0f50-xw5xh"
Oct 30 18:55:05.519336 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.519279    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1e6fa3-8aac-4733-8e3a-a1417428e1cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac2a0f50-xw5xh\" (UID: \"2a1e6fa3-8aac-4733-8e3a-a1417428e1cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2a0f50-xw5xh"
Oct 30 18:55:05.573156 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.573123    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk9gg\" (UniqueName: \"kubernetes.io/projected/2a1e6fa3-8aac-4733-8e3a-a1417428e1cb-kube-api-access-fk9gg\") pod \"maas-api-revoke-sub-ac2a0f50-xw5xh\" (UID: \"2a1e6fa3-8aac-4733-8e3a-a1417428e1cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2a0f50-xw5xh"
Oct 30 18:55:05.668976 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.668743    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:55:05.679324 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.679285    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2a0f50-xw5xh"
Oct 30 18:55:05.703424 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:05.703392    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a7b9ed6-8tvfd"]
Oct 30 18:55:05.708049 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:55:05.708022    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca24639a_0332_4c5d_94a8_c7d8b15c6602.slice/crio-5db62588da8d7528eb988e2aff2ccd4a40d55d3016eb7623bf75562f2af18002 WatchSource:0}: Error finding container 5db62588da8d7528eb988e2aff2ccd4a40d55d3016eb7623bf75562f2af18002: Status 404 returned error can't find the container with id 5db62588da8d7528eb988e2aff2ccd4a40d55d3016eb7623bf75562f2af18002
Oct 30 18:55:06.055377 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.055353    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac2a0f50-xw5xh"]
Oct 30 18:55:06.058552 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:55:06.058521    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1e6fa3_8aac_4733_8e3a_a1417428e1cb.slice/crio-80611afe98d5dc3ffc0d94e90e141b757dccba968740e194c99df547047cceca WatchSource:0}: Error finding container 80611afe98d5dc3ffc0d94e90e141b757dccba968740e194c99df547047cceca: Status 404 returned error can't find the container with id 80611afe98d5dc3ffc0d94e90e141b757dccba968740e194c99df547047cceca
Oct 30 18:55:06.244364 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.244340    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:55:06.283420 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.283391    4924 generic.go:356] "Generic (PLEG): container finished" podID="1e919a72-c6c1-4c1f-9e24-f7c5910021de" containerID="361e8e91eb1c6cbbde4411a2bb7a06085f708e45d17b5facf349fa8c5b6e250e" exitCode=0
Oct 30 18:55:06.283605 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.283440    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"
Oct 30 18:55:06.283605 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.283483    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn" event={"ID":"1e919a72-c6c1-4c1f-9e24-f7c5910021de","Type":"ContainerDied","Data":"361e8e91eb1c6cbbde4411a2bb7a06085f708e45d17b5facf349fa8c5b6e250e"}
Oct 30 18:55:06.283605 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.283538    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn" event={"ID":"1e919a72-c6c1-4c1f-9e24-f7c5910021de","Type":"ContainerDied","Data":"ac2c27ae02cb0fa8acbdfa537b614eb5fd0936912c639d4aa3ac4dec1a35898f"}
Oct 30 18:55:06.283605 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.283572    4924 scope.go:122] "RemoveContainer" containerID="361e8e91eb1c6cbbde4411a2bb7a06085f708e45d17b5facf349fa8c5b6e250e"
Oct 30 18:55:06.285093 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.285034    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7b9ed6-8tvfd" event={"ID":"ca24639a-0332-4c5d-94a8-c7d8b15c6602","Type":"ContainerStarted","Data":"eda4d14392e7feefc0e46aeb448447470c07b977d5df707bbe3b44807b4e519a"}
Oct 30 18:55:06.285218 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.285109    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7b9ed6-8tvfd" event={"ID":"ca24639a-0332-4c5d-94a8-c7d8b15c6602","Type":"ContainerStarted","Data":"5db62588da8d7528eb988e2aff2ccd4a40d55d3016eb7623bf75562f2af18002"}
Oct 30 18:55:06.286392 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.286373    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2a0f50-xw5xh" event={"ID":"2a1e6fa3-8aac-4733-8e3a-a1417428e1cb","Type":"ContainerStarted","Data":"80611afe98d5dc3ffc0d94e90e141b757dccba968740e194c99df547047cceca"}
Oct 30 18:55:06.293296 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.293274    4924 scope.go:122] "RemoveContainer" containerID="361e8e91eb1c6cbbde4411a2bb7a06085f708e45d17b5facf349fa8c5b6e250e"
Oct 30 18:55:06.293539 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:55:06.293520    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"361e8e91eb1c6cbbde4411a2bb7a06085f708e45d17b5facf349fa8c5b6e250e\": container with ID starting with 361e8e91eb1c6cbbde4411a2bb7a06085f708e45d17b5facf349fa8c5b6e250e not found: ID does not exist" containerID="361e8e91eb1c6cbbde4411a2bb7a06085f708e45d17b5facf349fa8c5b6e250e"
Oct 30 18:55:06.293606 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.293548    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361e8e91eb1c6cbbde4411a2bb7a06085f708e45d17b5facf349fa8c5b6e250e"} err="failed to get container status \"361e8e91eb1c6cbbde4411a2bb7a06085f708e45d17b5facf349fa8c5b6e250e\": rpc error: code = NotFound desc = could not find container \"361e8e91eb1c6cbbde4411a2bb7a06085f708e45d17b5facf349fa8c5b6e250e\": container with ID starting with 361e8e91eb1c6cbbde4411a2bb7a06085f708e45d17b5facf349fa8c5b6e250e not found: ID does not exist"
Oct 30 18:55:06.325747 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.325725    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e919a72-c6c1-4c1f-9e24-f7c5910021de-kube-api-access-psp9w\" (UniqueName: \"kubernetes.io/projected/1e919a72-c6c1-4c1f-9e24-f7c5910021de-kube-api-access-psp9w\") pod \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") "
Oct 30 18:55:06.325859 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.325760    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-token\" (UniqueName: \"kubernetes.io/projected/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-token\") pod \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") "
Oct 30 18:55:06.325859 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.325787    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-workload-certs\") pod \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") "
Oct 30 18:55:06.325859 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.325820    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istiod-ca-cert\") pod \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") "
Oct 30 18:55:06.325987 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.325915    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-envoy\") pod \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") "
Oct 30 18:55:06.326047 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.326033    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-data\") pod \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") "
Oct 30 18:55:06.326147 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.326081    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-podinfo\") pod \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") "
Oct 30 18:55:06.326147 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.326134    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-workload-socket\") pod \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") "
Oct 30 18:55:06.326268 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.326177    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-workload-certs" pod "1e919a72-c6c1-4c1f-9e24-f7c5910021de" (UID: "1e919a72-c6c1-4c1f-9e24-f7c5910021de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:06.326268 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.326192    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-credential-socket\") pod \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\" (UID: \"1e919a72-c6c1-4c1f-9e24-f7c5910021de\") "
Oct 30 18:55:06.326591 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.326304    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istiod-ca-cert" pod "1e919a72-c6c1-4c1f-9e24-f7c5910021de" (UID: "1e919a72-c6c1-4c1f-9e24-f7c5910021de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:06.326591 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.326395    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-workload-socket" pod "1e919a72-c6c1-4c1f-9e24-f7c5910021de" (UID: "1e919a72-c6c1-4c1f-9e24-f7c5910021de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:06.326591 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.326563    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-credential-socket" pod "1e919a72-c6c1-4c1f-9e24-f7c5910021de" (UID: "1e919a72-c6c1-4c1f-9e24-f7c5910021de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:06.326591 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.326578    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-data" pod "1e919a72-c6c1-4c1f-9e24-f7c5910021de" (UID: "1e919a72-c6c1-4c1f-9e24-f7c5910021de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:06.326871 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.326849    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:06.326934 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.326876    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:06.326934 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.326890    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:06.326934 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.326905    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:06.326934 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.326918    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:06.328485 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.328459    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-token" pod "1e919a72-c6c1-4c1f-9e24-f7c5910021de" (UID: "1e919a72-c6c1-4c1f-9e24-f7c5910021de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:06.328576 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.328493    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e919a72-c6c1-4c1f-9e24-f7c5910021de-kube-api-access-psp9w" pod "1e919a72-c6c1-4c1f-9e24-f7c5910021de" (UID: "1e919a72-c6c1-4c1f-9e24-f7c5910021de"). InnerVolumeSpecName "kube-api-access-psp9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:06.328576 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.328544    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-envoy" pod "1e919a72-c6c1-4c1f-9e24-f7c5910021de" (UID: "1e919a72-c6c1-4c1f-9e24-f7c5910021de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:55:06.328576 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.328560    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-podinfo" pod "1e919a72-c6c1-4c1f-9e24-f7c5910021de" (UID: "1e919a72-c6c1-4c1f-9e24-f7c5910021de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:55:06.428713 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.428672    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:06.428713 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.428717    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psp9w\" (UniqueName: \"kubernetes.io/projected/1e919a72-c6c1-4c1f-9e24-f7c5910021de-kube-api-access-psp9w\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:06.428929 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.428735    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:06.428929 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.428749    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e919a72-c6c1-4c1f-9e24-f7c5910021de-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:06.467688 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.467579    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7b9ed6-8tvfd" podStartSLOduration=2.4675642509999998 podStartE2EDuration="2.467564251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:06.464438979 +0000 UTC m=+1107.741489423" watchObservedRunningTime="2026-10-30 18:55:06.467564251 +0000 UTC m=+1107.744614696"
Oct 30 18:55:06.701350 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.701316    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"]
Oct 30 18:55:06.749873 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.749841    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f6zwn"]
Oct 30 18:55:06.834755 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:06.834726    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e919a72-c6c1-4c1f-9e24-f7c5910021de" path="/var/lib/kubelet/pods/1e919a72-c6c1-4c1f-9e24-f7c5910021de/volumes"
Oct 30 18:55:07.295492 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:07.295460    4924 generic.go:356] "Generic (PLEG): container finished" podID="ca24639a-0332-4c5d-94a8-c7d8b15c6602" containerID="eda4d14392e7feefc0e46aeb448447470c07b977d5df707bbe3b44807b4e519a" exitCode=0
Oct 30 18:55:07.295932 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:07.295512    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7b9ed6-8tvfd" event={"ID":"ca24639a-0332-4c5d-94a8-c7d8b15c6602","Type":"ContainerDied","Data":"eda4d14392e7feefc0e46aeb448447470c07b977d5df707bbe3b44807b4e519a"}
Oct 30 18:55:07.296949 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:07.296927    4924 generic.go:356] "Generic (PLEG): container finished" podID="2a1e6fa3-8aac-4733-8e3a-a1417428e1cb" containerID="2d9bc7e2db3937e94b4309f2ffc3f3a391af48946d94eadb27681a015533e874" exitCode=0
Oct 30 18:55:07.297056 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:07.297012    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2a0f50-xw5xh" event={"ID":"2a1e6fa3-8aac-4733-8e3a-a1417428e1cb","Type":"ContainerDied","Data":"2d9bc7e2db3937e94b4309f2ffc3f3a391af48946d94eadb27681a015533e874"}
Oct 30 18:55:07.789281 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:07.789251    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:55:07.801527 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:07.801501    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:55:08.322356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:08.322325    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:55:08.502564 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:08.502536    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2a0f50-xw5xh"
Oct 30 18:55:08.504590 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:08.504572    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7b9ed6-8tvfd"
Oct 30 18:55:08.556253 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:08.556224    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a1e6fa3-8aac-4733-8e3a-a1417428e1cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1e6fa3-8aac-4733-8e3a-a1417428e1cb-maas-api-xks-ca\") pod \"2a1e6fa3-8aac-4733-8e3a-a1417428e1cb\" (UID: \"2a1e6fa3-8aac-4733-8e3a-a1417428e1cb\") "
Oct 30 18:55:08.556412 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:08.556304    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1e6fa3-8aac-4733-8e3a-a1417428e1cb-kube-api-access-fk9gg\" (UniqueName: \"kubernetes.io/projected/2a1e6fa3-8aac-4733-8e3a-a1417428e1cb-kube-api-access-fk9gg\") pod \"2a1e6fa3-8aac-4733-8e3a-a1417428e1cb\" (UID: \"2a1e6fa3-8aac-4733-8e3a-a1417428e1cb\") "
Oct 30 18:55:08.556412 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:08.556351    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca24639a-0332-4c5d-94a8-c7d8b15c6602-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca24639a-0332-4c5d-94a8-c7d8b15c6602-maas-api-xks-ca\") pod \"ca24639a-0332-4c5d-94a8-c7d8b15c6602\" (UID: \"ca24639a-0332-4c5d-94a8-c7d8b15c6602\") "
Oct 30 18:55:08.556412 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:08.556382    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a1e6fa3-8aac-4733-8e3a-a1417428e1cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1e6fa3-8aac-4733-8e3a-a1417428e1cb-maas-api-openshift-ca\") pod \"2a1e6fa3-8aac-4733-8e3a-a1417428e1cb\" (UID: \"2a1e6fa3-8aac-4733-8e3a-a1417428e1cb\") "
Oct 30 18:55:08.556545 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:08.556444    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca24639a-0332-4c5d-94a8-c7d8b15c6602-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca24639a-0332-4c5d-94a8-c7d8b15c6602-maas-api-openshift-ca\") pod \"ca24639a-0332-4c5d-94a8-c7d8b15c6602\" (UID: \"ca24639a-0332-4c5d-94a8-c7d8b15c6602\") "
Oct 30 18:55:08.556545 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:08.556502    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca24639a-0332-4c5d-94a8-c7d8b15c6602-kube-api-access-lbqgc\" (UniqueName: \"kubernetes.io/projected/ca24639a-0332-4c5d-94a8-c7d8b15c6602-kube-api-access-lbqgc\") pod \"ca24639a-0332-4c5d-94a8-c7d8b15c6602\" (UID: \"ca24639a-0332-4c5d-94a8-c7d8b15c6602\") "
Oct 30 18:55:08.556802 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:08.556770    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1e6fa3-8aac-4733-8e3a-a1417428e1cb-maas-api-openshift-ca" pod "2a1e6fa3-8aac-4733-8e3a-a1417428e1cb" (UID: "2a1e6fa3-8aac-4733-8e3a-a1417428e1cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:08.556949 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:08.556856    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca24639a-0332-4c5d-94a8-c7d8b15c6602-maas-api-openshift-ca" pod "ca24639a-0332-4c5d-94a8-c7d8b15c6602" (UID: "ca24639a-0332-4c5d-94a8-c7d8b15c6602"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:08.557038 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:08.557012    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca24639a-0332-4c5d-94a8-c7d8b15c6602-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:08.557170 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:08.557040    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1e6fa3-8aac-4733-8e3a-a1417428e1cb-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:08.558494 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:08.558470    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1e6fa3-8aac-4733-8e3a-a1417428e1cb-maas-api-xks-ca" pod "2a1e6fa3-8aac-4733-8e3a-a1417428e1cb" (UID: "2a1e6fa3-8aac-4733-8e3a-a1417428e1cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:08.558494 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:08.558481    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1e6fa3-8aac-4733-8e3a-a1417428e1cb-kube-api-access-fk9gg" pod "2a1e6fa3-8aac-4733-8e3a-a1417428e1cb" (UID: "2a1e6fa3-8aac-4733-8e3a-a1417428e1cb"). InnerVolumeSpecName "kube-api-access-fk9gg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:08.558618 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:08.558541    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca24639a-0332-4c5d-94a8-c7d8b15c6602-maas-api-xks-ca" pod "ca24639a-0332-4c5d-94a8-c7d8b15c6602" (UID: "ca24639a-0332-4c5d-94a8-c7d8b15c6602"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:08.558684 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:08.558656    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca24639a-0332-4c5d-94a8-c7d8b15c6602-kube-api-access-lbqgc" pod "ca24639a-0332-4c5d-94a8-c7d8b15c6602" (UID: "ca24639a-0332-4c5d-94a8-c7d8b15c6602"). InnerVolumeSpecName "kube-api-access-lbqgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:08.658439 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:08.658407    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbqgc\" (UniqueName: \"kubernetes.io/projected/ca24639a-0332-4c5d-94a8-c7d8b15c6602-kube-api-access-lbqgc\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:08.658439 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:08.658433    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1e6fa3-8aac-4733-8e3a-a1417428e1cb-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:08.658439 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:08.658445    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk9gg\" (UniqueName: \"kubernetes.io/projected/2a1e6fa3-8aac-4733-8e3a-a1417428e1cb-kube-api-access-fk9gg\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:08.658654 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:08.658454    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca24639a-0332-4c5d-94a8-c7d8b15c6602-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:09.317389 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:09.317360    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7b9ed6-8tvfd"
Oct 30 18:55:09.317389 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:09.317385    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7b9ed6-8tvfd" event={"ID":"ca24639a-0332-4c5d-94a8-c7d8b15c6602","Type":"ContainerDied","Data":"5db62588da8d7528eb988e2aff2ccd4a40d55d3016eb7623bf75562f2af18002"}
Oct 30 18:55:09.317661 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:09.317431    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5db62588da8d7528eb988e2aff2ccd4a40d55d3016eb7623bf75562f2af18002"
Oct 30 18:55:09.318812 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:09.318786    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2a0f50-xw5xh"
Oct 30 18:55:09.318927 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:09.318812    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2a0f50-xw5xh" event={"ID":"2a1e6fa3-8aac-4733-8e3a-a1417428e1cb","Type":"ContainerDied","Data":"80611afe98d5dc3ffc0d94e90e141b757dccba968740e194c99df547047cceca"}
Oct 30 18:55:09.318927 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:09.318846    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80611afe98d5dc3ffc0d94e90e141b757dccba968740e194c99df547047cceca"
Oct 30 18:55:10.512157 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:10.512124    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:55:13.189974 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.189940    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-725a884d-nk6zp"]
Oct 30 18:55:13.190458 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.190444    4924 cpu_manager.go:414] "containerMap: removing container" podUID="1e919a72-c6c1-4c1f-9e24-f7c5910021de" containerName="istio-proxy"
Oct 30 18:55:13.190511 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.190458    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e919a72-c6c1-4c1f-9e24-f7c5910021de" containerName="istio-proxy"
Oct 30 18:55:13.190511 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.190484    4924 cpu_manager.go:414] "containerMap: removing container" podUID="ca24639a-0332-4c5d-94a8-c7d8b15c6602" containerName="revoke-keys"
Oct 30 18:55:13.190511 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.190490    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca24639a-0332-4c5d-94a8-c7d8b15c6602" containerName="revoke-keys"
Oct 30 18:55:13.190511 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.190498    4924 cpu_manager.go:414] "containerMap: removing container" podUID="2a1e6fa3-8aac-4733-8e3a-a1417428e1cb" containerName="revoke-keys"
Oct 30 18:55:13.190511 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.190502    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1e6fa3-8aac-4733-8e3a-a1417428e1cb" containerName="revoke-keys"
Oct 30 18:55:13.190668 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.190591    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e919a72-c6c1-4c1f-9e24-f7c5910021de" containerName="istio-proxy"
Oct 30 18:55:13.190668 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.190601    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca24639a-0332-4c5d-94a8-c7d8b15c6602" containerName="revoke-keys"
Oct 30 18:55:13.190668 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.190607    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1e6fa3-8aac-4733-8e3a-a1417428e1cb" containerName="revoke-keys"
Oct 30 18:55:13.195222 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.195206    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725a884d-nk6zp"
Oct 30 18:55:13.248084 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.248036    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-725a884d-nk6zp"]
Oct 30 18:55:13.290902 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.290870    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:55:13.290902 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.290879    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:55:13.291151 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.290921    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:55:13.291151 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.291147    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 18:55:13.302781 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.302751    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6720c96-91cf-48d3-8f67-41952fe0f88e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-725a884d-nk6zp\" (UID: \"c6720c96-91cf-48d3-8f67-41952fe0f88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725a884d-nk6zp"
Oct 30 18:55:13.302919 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.302892    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqx9f\" (UniqueName: \"kubernetes.io/projected/c6720c96-91cf-48d3-8f67-41952fe0f88e-kube-api-access-pqx9f\") pod \"maas-api-revoke-sub-725a884d-nk6zp\" (UID: \"c6720c96-91cf-48d3-8f67-41952fe0f88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725a884d-nk6zp"
Oct 30 18:55:13.303207 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.303181    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6720c96-91cf-48d3-8f67-41952fe0f88e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-725a884d-nk6zp\" (UID: \"c6720c96-91cf-48d3-8f67-41952fe0f88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725a884d-nk6zp"
Oct 30 18:55:13.404731 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.404699    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6720c96-91cf-48d3-8f67-41952fe0f88e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-725a884d-nk6zp\" (UID: \"c6720c96-91cf-48d3-8f67-41952fe0f88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725a884d-nk6zp"
Oct 30 18:55:13.404903 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.404747    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6720c96-91cf-48d3-8f67-41952fe0f88e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-725a884d-nk6zp\" (UID: \"c6720c96-91cf-48d3-8f67-41952fe0f88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725a884d-nk6zp"
Oct 30 18:55:13.404903 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.404798    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqx9f\" (UniqueName: \"kubernetes.io/projected/c6720c96-91cf-48d3-8f67-41952fe0f88e-kube-api-access-pqx9f\") pod \"maas-api-revoke-sub-725a884d-nk6zp\" (UID: \"c6720c96-91cf-48d3-8f67-41952fe0f88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725a884d-nk6zp"
Oct 30 18:55:13.405424 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.405401    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6720c96-91cf-48d3-8f67-41952fe0f88e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-725a884d-nk6zp\" (UID: \"c6720c96-91cf-48d3-8f67-41952fe0f88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725a884d-nk6zp"
Oct 30 18:55:13.407079 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.407044    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6720c96-91cf-48d3-8f67-41952fe0f88e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-725a884d-nk6zp\" (UID: \"c6720c96-91cf-48d3-8f67-41952fe0f88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725a884d-nk6zp"
Oct 30 18:55:13.459530 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.459471    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqx9f\" (UniqueName: \"kubernetes.io/projected/c6720c96-91cf-48d3-8f67-41952fe0f88e-kube-api-access-pqx9f\") pod \"maas-api-revoke-sub-725a884d-nk6zp\" (UID: \"c6720c96-91cf-48d3-8f67-41952fe0f88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725a884d-nk6zp"
Oct 30 18:55:13.507854 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.507824    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725a884d-nk6zp"
Oct 30 18:55:13.878313 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:13.878277    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-725a884d-nk6zp"]
Oct 30 18:55:13.878628 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:55:13.878605    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6720c96_91cf_48d3_8f67_41952fe0f88e.slice/crio-8b3e873ddfb6986b8891519e8a43016aa333bdbf9a7b5f5e9534216b3c29f9ef WatchSource:0}: Error finding container 8b3e873ddfb6986b8891519e8a43016aa333bdbf9a7b5f5e9534216b3c29f9ef: Status 404 returned error can't find the container with id 8b3e873ddfb6986b8891519e8a43016aa333bdbf9a7b5f5e9534216b3c29f9ef
Oct 30 18:55:14.350661 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:14.350623    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725a884d-nk6zp" event={"ID":"c6720c96-91cf-48d3-8f67-41952fe0f88e","Type":"ContainerStarted","Data":"1081407cf428e398687d8b0fcb479d6ca08567869558c803deb5797f9f2bc315"}
Oct 30 18:55:14.351018 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:14.350682    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725a884d-nk6zp" event={"ID":"c6720c96-91cf-48d3-8f67-41952fe0f88e","Type":"ContainerStarted","Data":"8b3e873ddfb6986b8891519e8a43016aa333bdbf9a7b5f5e9534216b3c29f9ef"}
Oct 30 18:55:14.461208 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:14.461105    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725a884d-nk6zp" podStartSLOduration=1.461091063 podStartE2EDuration="1.461091063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:14.458163904 +0000 UTC m=+1115.735214350" watchObservedRunningTime="2026-10-30 18:55:14.461091063 +0000 UTC m=+1115.738141507"
Oct 30 18:55:15.249688 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:15.249639    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:55:15.383693 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:15.383660    4924 generic.go:356] "Generic (PLEG): container finished" podID="c6720c96-91cf-48d3-8f67-41952fe0f88e" containerID="1081407cf428e398687d8b0fcb479d6ca08567869558c803deb5797f9f2bc315" exitCode=0
Oct 30 18:55:15.384341 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:15.384314    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725a884d-nk6zp" event={"ID":"c6720c96-91cf-48d3-8f67-41952fe0f88e","Type":"ContainerDied","Data":"1081407cf428e398687d8b0fcb479d6ca08567869558c803deb5797f9f2bc315"}
Oct 30 18:55:15.714804 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:55:15.714767    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:55:16.556310 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:16.556287    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725a884d-nk6zp"
Oct 30 18:55:16.663511 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:16.663403    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6720c96-91cf-48d3-8f67-41952fe0f88e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6720c96-91cf-48d3-8f67-41952fe0f88e-maas-api-openshift-ca\") pod \"c6720c96-91cf-48d3-8f67-41952fe0f88e\" (UID: \"c6720c96-91cf-48d3-8f67-41952fe0f88e\") "
Oct 30 18:55:16.663511 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:16.663474    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6720c96-91cf-48d3-8f67-41952fe0f88e-kube-api-access-pqx9f\" (UniqueName: \"kubernetes.io/projected/c6720c96-91cf-48d3-8f67-41952fe0f88e-kube-api-access-pqx9f\") pod \"c6720c96-91cf-48d3-8f67-41952fe0f88e\" (UID: \"c6720c96-91cf-48d3-8f67-41952fe0f88e\") "
Oct 30 18:55:16.663716 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:16.663533    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6720c96-91cf-48d3-8f67-41952fe0f88e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6720c96-91cf-48d3-8f67-41952fe0f88e-maas-api-xks-ca\") pod \"c6720c96-91cf-48d3-8f67-41952fe0f88e\" (UID: \"c6720c96-91cf-48d3-8f67-41952fe0f88e\") "
Oct 30 18:55:16.663867 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:16.663836    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6720c96-91cf-48d3-8f67-41952fe0f88e-maas-api-openshift-ca" pod "c6720c96-91cf-48d3-8f67-41952fe0f88e" (UID: "c6720c96-91cf-48d3-8f67-41952fe0f88e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:16.664104 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:16.664084    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6720c96-91cf-48d3-8f67-41952fe0f88e-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:16.666010 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:16.665955    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6720c96-91cf-48d3-8f67-41952fe0f88e-kube-api-access-pqx9f" pod "c6720c96-91cf-48d3-8f67-41952fe0f88e" (UID: "c6720c96-91cf-48d3-8f67-41952fe0f88e"). InnerVolumeSpecName "kube-api-access-pqx9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:16.666200 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:16.665971    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6720c96-91cf-48d3-8f67-41952fe0f88e-maas-api-xks-ca" pod "c6720c96-91cf-48d3-8f67-41952fe0f88e" (UID: "c6720c96-91cf-48d3-8f67-41952fe0f88e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:16.765658 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:16.765629    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqx9f\" (UniqueName: \"kubernetes.io/projected/c6720c96-91cf-48d3-8f67-41952fe0f88e-kube-api-access-pqx9f\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:16.765658 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:16.765658    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6720c96-91cf-48d3-8f67-41952fe0f88e-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:17.397940 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:17.397910    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725a884d-nk6zp" event={"ID":"c6720c96-91cf-48d3-8f67-41952fe0f88e","Type":"ContainerDied","Data":"8b3e873ddfb6986b8891519e8a43016aa333bdbf9a7b5f5e9534216b3c29f9ef"}
Oct 30 18:55:17.398175 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:17.397943    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725a884d-nk6zp"
Oct 30 18:55:17.398175 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:17.397956    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b3e873ddfb6986b8891519e8a43016aa333bdbf9a7b5f5e9534216b3c29f9ef"
Oct 30 18:55:20.216040 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:20.215961    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:55:23.068661 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:23.068622    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:55:24.969770 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:24.969736    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:55:25.624825 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:25.624798    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:55:25.807014 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:25.806982    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:55:29.757098 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:29.757044    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:55:34.506394 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:34.506355    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:55:34.678514 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:34.678486    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:55:39.214812 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:39.214761    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:55:39.875612 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:39.875580    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"]
Oct 30 18:55:39.876137 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:39.876123    4924 cpu_manager.go:414] "containerMap: removing container" podUID="c6720c96-91cf-48d3-8f67-41952fe0f88e" containerName="revoke-keys"
Oct 30 18:55:39.876190 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:39.876139    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6720c96-91cf-48d3-8f67-41952fe0f88e" containerName="revoke-keys"
Oct 30 18:55:39.876264 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:39.876254    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6720c96-91cf-48d3-8f67-41952fe0f88e" containerName="revoke-keys"
Oct 30 18:55:39.881019 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:39.881000    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:39.908594 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:39.908563    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-data\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:39.908704 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:39.908612    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-credential-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:39.908704 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:39.908684    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-envoy\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:39.908832 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:39.908811    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d743094-1851-4c3a-9866-ecb1062c69b6-istiod-ca-cert\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:39.908888 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:39.908844    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzqx9\" (UniqueName: \"kubernetes.io/projected/7d743094-1851-4c3a-9866-ecb1062c69b6-kube-api-access-bzqx9\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:39.908888 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:39.908865    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-workload-certs\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:39.909084 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:39.909048    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-podinfo\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:39.909157 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:39.909091    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-workload-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:39.909157 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:39.909123    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-token\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:39.980936 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:39.980854    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"]
Oct 30 18:55:40.009654 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.009630    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-podinfo\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:40.009793 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.009661    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-workload-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:40.009793 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.009695    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-token\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:40.009793 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.009723    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-data\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:40.009793 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.009766    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-credential-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:40.009793 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.009793    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-envoy\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:40.010085 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.009883    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d743094-1851-4c3a-9866-ecb1062c69b6-istiod-ca-cert\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:40.010085 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.009913    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzqx9\" (UniqueName: \"kubernetes.io/projected/7d743094-1851-4c3a-9866-ecb1062c69b6-kube-api-access-bzqx9\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:40.010085 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.009947    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-workload-certs\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:40.010263 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.010145    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-workload-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:40.010361 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.010336    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-data\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:40.010514 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.010488    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-workload-certs\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:40.011027 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.011003    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d743094-1851-4c3a-9866-ecb1062c69b6-istiod-ca-cert\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:40.011371 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.011352    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-credential-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:40.012289 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.012267    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-podinfo\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:40.012623 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.012598    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-envoy\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:40.063625 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.063595    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-token\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:40.063792 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.063772    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzqx9\" (UniqueName: \"kubernetes.io/projected/7d743094-1851-4c3a-9866-ecb1062c69b6-kube-api-access-bzqx9\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:40.193995 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.193916    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:40.398015 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.397985    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"]
Oct 30 18:55:40.402401 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.402380    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.458730 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.458659    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"]
Oct 30 18:55:40.515253 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.515219    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-workload-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.515418 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.515273    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-token\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.515418 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.515410    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-data\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.515491 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.515437    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-podinfo\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.515491 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.515475    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-credential-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.515569 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.515532    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-envoy\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.515685 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.515665    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k27wb\" (UniqueName: \"kubernetes.io/projected/816f124c-a4bb-478e-b48a-c8b91a4113d2-kube-api-access-k27wb\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.515747 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.515712    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-workload-certs\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.515747 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.515729    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/816f124c-a4bb-478e-b48a-c8b91a4113d2-istiod-ca-cert\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.611867 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.611832    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"]
Oct 30 18:55:40.612102 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:55:40.612075    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d743094_1851_4c3a_9866_ecb1062c69b6.slice/crio-ad409711193beda63574b25f5581d1d6069e3397fca3a76bd136570e326e0165 WatchSource:0}: Error finding container ad409711193beda63574b25f5581d1d6069e3397fca3a76bd136570e326e0165: Status 404 returned error can't find the container with id ad409711193beda63574b25f5581d1d6069e3397fca3a76bd136570e326e0165
Oct 30 18:55:40.614470 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.614442    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:55:40.614544 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.614499    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:55:40.614544 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.614527    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:55:40.616511 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.616440    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-data\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.616511 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.616472    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-podinfo\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.616511 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.616503    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-credential-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.616689 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.616663    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-envoy\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.616738 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.616711    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k27wb\" (UniqueName: \"kubernetes.io/projected/816f124c-a4bb-478e-b48a-c8b91a4113d2-kube-api-access-k27wb\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.616796 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.616745    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-workload-certs\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.616796 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.616772    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/816f124c-a4bb-478e-b48a-c8b91a4113d2-istiod-ca-cert\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.616910 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.616857    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-workload-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.616910 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.616874    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-data\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.617026 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.616937    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-token\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.617210 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.617183    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-workload-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.617448 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.617424    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-credential-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.617448 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.617438    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-workload-certs\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.617730 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.617711    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/816f124c-a4bb-478e-b48a-c8b91a4113d2-istiod-ca-cert\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.618926 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.618907    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-envoy\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.619034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.619020    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-podinfo\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.671854 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.671825    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k27wb\" (UniqueName: \"kubernetes.io/projected/816f124c-a4bb-478e-b48a-c8b91a4113d2-kube-api-access-k27wb\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.672184 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.672165    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-token\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:40.716247 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:40.716150    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:41.141663 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:55:41.141630    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod816f124c_a4bb_478e_b48a_c8b91a4113d2.slice/crio-033debdce115fe8da3135b8a72ad22215736a208482481be11bdaa40a777b195 WatchSource:0}: Error finding container 033debdce115fe8da3135b8a72ad22215736a208482481be11bdaa40a777b195: Status 404 returned error can't find the container with id 033debdce115fe8da3135b8a72ad22215736a208482481be11bdaa40a777b195
Oct 30 18:55:41.142044 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:41.142023    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"]
Oct 30 18:55:41.144241 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:41.144210    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:55:41.144362 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:41.144268    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:55:41.144362 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:41.144297    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:55:41.550199 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:41.550105    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn" event={"ID":"816f124c-a4bb-478e-b48a-c8b91a4113d2","Type":"ContainerStarted","Data":"8e47758cc8abe73a548256eb8bca57e4ad7e7be1e983349a6150169d6ddb2a3a"}
Oct 30 18:55:41.550199 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:41.550166    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn" event={"ID":"816f124c-a4bb-478e-b48a-c8b91a4113d2","Type":"ContainerStarted","Data":"033debdce115fe8da3135b8a72ad22215736a208482481be11bdaa40a777b195"}
Oct 30 18:55:41.552080 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:41.552044    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk" event={"ID":"7d743094-1851-4c3a-9866-ecb1062c69b6","Type":"ContainerStarted","Data":"d898f5123e81a57cd207d07ece648447a1cb0b5e632a8f96ce1333f52d89d449"}
Oct 30 18:55:41.552210 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:41.552114    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk" event={"ID":"7d743094-1851-4c3a-9866-ecb1062c69b6","Type":"ContainerStarted","Data":"ad409711193beda63574b25f5581d1d6069e3397fca3a76bd136570e326e0165"}
Oct 30 18:55:41.663568 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:41.663512    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn" podStartSLOduration=1.663499182 podStartE2EDuration="1.663499182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:41.662042846 +0000 UTC m=+1142.939093295" watchObservedRunningTime="2026-10-30 18:55:41.663499182 +0000 UTC m=+1142.940549627"
Oct 30 18:55:41.724085 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:41.723403    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:41.787101 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:41.785987    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk" podStartSLOduration=2.785968077 podStartE2EDuration="2.785968077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:41.784699013 +0000 UTC m=+1143.061749459" watchObservedRunningTime="2026-10-30 18:55:41.785968077 +0000 UTC m=+1143.063018523"
Oct 30 18:55:42.194226 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:42.194193    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:42.196098 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:42.196074    4924 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 18:55:42.196169 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:42.196119    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk" podUID="7d743094-1851-4c3a-9866-ecb1062c69b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 18:55:42.723089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:42.721870    4924 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:55:42.723089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:42.721935    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn" podUID="816f124c-a4bb-478e-b48a-c8b91a4113d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 18:55:43.195167 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:43.195132    4924 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 18:55:43.195349 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:43.195198    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk" podUID="7d743094-1851-4c3a-9866-ecb1062c69b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 18:55:43.722507 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:43.722469    4924 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:55:43.722807 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:43.722775    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn" podUID="816f124c-a4bb-478e-b48a-c8b91a4113d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 18:55:44.031518 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:44.028796    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:55:44.194858 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:44.194826    4924 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 18:55:44.195091 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:44.194877    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk" podUID="7d743094-1851-4c3a-9866-ecb1062c69b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 18:55:44.724088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:44.718605    4924 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:55:44.724088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:44.718680    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn" podUID="816f124c-a4bb-478e-b48a-c8b91a4113d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 18:55:45.194837 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:45.194804    4924 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 18:55:45.195307 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:45.194851    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk" podUID="7d743094-1851-4c3a-9866-ecb1062c69b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 18:55:45.725485 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:45.725441    4924 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:55:45.725755 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:45.725724    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn" podUID="816f124c-a4bb-478e-b48a-c8b91a4113d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 18:55:45.772222 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:55:45.772185    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:55:46.195017 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.194987    4924 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 18:55:46.195489 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.195139    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk" podUID="7d743094-1851-4c3a-9866-ecb1062c69b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 18:55:46.496076 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.495990    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fdecb3b-zpc44"]
Oct 30 18:55:46.542692 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.522049    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fdecb3b-zpc44"
Oct 30 18:55:46.571233 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.571205    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e280a1e-pjbp7"]
Oct 30 18:55:46.578173 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.576966    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fdecb3b-zpc44"]
Oct 30 18:55:46.578173 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.577101    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e280a1e-pjbp7"
Oct 30 18:55:46.594878 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.594851    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:55:46.594995 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.594962    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 18:55:46.597181 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.597157    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e280a1e-pjbp7"]
Oct 30 18:55:46.601603 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.601584    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:55:46.601603 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.601583    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:55:46.640088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.626764    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a267a84-7dc4-4098-a7d3-4b0e8fb1867e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fdecb3b-zpc44\" (UID: \"9a267a84-7dc4-4098-a7d3-4b0e8fb1867e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fdecb3b-zpc44"
Oct 30 18:55:46.640088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.626871    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a267a84-7dc4-4098-a7d3-4b0e8fb1867e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fdecb3b-zpc44\" (UID: \"9a267a84-7dc4-4098-a7d3-4b0e8fb1867e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fdecb3b-zpc44"
Oct 30 18:55:46.640088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.626906    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4snt7\" (UniqueName: \"kubernetes.io/projected/9a267a84-7dc4-4098-a7d3-4b0e8fb1867e-kube-api-access-4snt7\") pod \"maas-api-revoke-sub-6fdecb3b-zpc44\" (UID: \"9a267a84-7dc4-4098-a7d3-4b0e8fb1867e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fdecb3b-zpc44"
Oct 30 18:55:46.640088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.626965    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a545e6e7-3f1f-42ed-8f96-147bef8d70c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e280a1e-pjbp7\" (UID: \"a545e6e7-3f1f-42ed-8f96-147bef8d70c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e280a1e-pjbp7"
Oct 30 18:55:46.640088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.627059    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a545e6e7-3f1f-42ed-8f96-147bef8d70c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e280a1e-pjbp7\" (UID: \"a545e6e7-3f1f-42ed-8f96-147bef8d70c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e280a1e-pjbp7"
Oct 30 18:55:46.640088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.627265    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lf54\" (UniqueName: \"kubernetes.io/projected/a545e6e7-3f1f-42ed-8f96-147bef8d70c6-kube-api-access-6lf54\") pod \"maas-api-revoke-sub-0e280a1e-pjbp7\" (UID: \"a545e6e7-3f1f-42ed-8f96-147bef8d70c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e280a1e-pjbp7"
Oct 30 18:55:46.719098 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.718428    4924 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:55:46.719098 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.718486    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn" podUID="816f124c-a4bb-478e-b48a-c8b91a4113d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 18:55:46.752683 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.752591    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lf54\" (UniqueName: \"kubernetes.io/projected/a545e6e7-3f1f-42ed-8f96-147bef8d70c6-kube-api-access-6lf54\") pod \"maas-api-revoke-sub-0e280a1e-pjbp7\" (UID: \"a545e6e7-3f1f-42ed-8f96-147bef8d70c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e280a1e-pjbp7"
Oct 30 18:55:46.752683 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.752654    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a267a84-7dc4-4098-a7d3-4b0e8fb1867e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fdecb3b-zpc44\" (UID: \"9a267a84-7dc4-4098-a7d3-4b0e8fb1867e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fdecb3b-zpc44"
Oct 30 18:55:46.752898 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.752735    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a267a84-7dc4-4098-a7d3-4b0e8fb1867e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fdecb3b-zpc44\" (UID: \"9a267a84-7dc4-4098-a7d3-4b0e8fb1867e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fdecb3b-zpc44"
Oct 30 18:55:46.752898 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.752771    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4snt7\" (UniqueName: \"kubernetes.io/projected/9a267a84-7dc4-4098-a7d3-4b0e8fb1867e-kube-api-access-4snt7\") pod \"maas-api-revoke-sub-6fdecb3b-zpc44\" (UID: \"9a267a84-7dc4-4098-a7d3-4b0e8fb1867e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fdecb3b-zpc44"
Oct 30 18:55:46.752898 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.752824    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a545e6e7-3f1f-42ed-8f96-147bef8d70c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e280a1e-pjbp7\" (UID: \"a545e6e7-3f1f-42ed-8f96-147bef8d70c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e280a1e-pjbp7"
Oct 30 18:55:46.753055 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.752916    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a545e6e7-3f1f-42ed-8f96-147bef8d70c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e280a1e-pjbp7\" (UID: \"a545e6e7-3f1f-42ed-8f96-147bef8d70c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e280a1e-pjbp7"
Oct 30 18:55:46.753779 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.753753    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a545e6e7-3f1f-42ed-8f96-147bef8d70c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e280a1e-pjbp7\" (UID: \"a545e6e7-3f1f-42ed-8f96-147bef8d70c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e280a1e-pjbp7"
Oct 30 18:55:46.753973 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.753931    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a267a84-7dc4-4098-a7d3-4b0e8fb1867e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fdecb3b-zpc44\" (UID: \"9a267a84-7dc4-4098-a7d3-4b0e8fb1867e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fdecb3b-zpc44"
Oct 30 18:55:46.755932 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.755910    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a267a84-7dc4-4098-a7d3-4b0e8fb1867e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fdecb3b-zpc44\" (UID: \"9a267a84-7dc4-4098-a7d3-4b0e8fb1867e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fdecb3b-zpc44"
Oct 30 18:55:46.756078 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.756037    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a545e6e7-3f1f-42ed-8f96-147bef8d70c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e280a1e-pjbp7\" (UID: \"a545e6e7-3f1f-42ed-8f96-147bef8d70c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e280a1e-pjbp7"
Oct 30 18:55:46.825848 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.825819    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lf54\" (UniqueName: \"kubernetes.io/projected/a545e6e7-3f1f-42ed-8f96-147bef8d70c6-kube-api-access-6lf54\") pod \"maas-api-revoke-sub-0e280a1e-pjbp7\" (UID: \"a545e6e7-3f1f-42ed-8f96-147bef8d70c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e280a1e-pjbp7"
Oct 30 18:55:46.859243 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.834237    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4snt7\" (UniqueName: \"kubernetes.io/projected/9a267a84-7dc4-4098-a7d3-4b0e8fb1867e-kube-api-access-4snt7\") pod \"maas-api-revoke-sub-6fdecb3b-zpc44\" (UID: \"9a267a84-7dc4-4098-a7d3-4b0e8fb1867e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fdecb3b-zpc44"
Oct 30 18:55:46.870690 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.870658    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fdecb3b-zpc44"
Oct 30 18:55:46.896560 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:46.896531    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e280a1e-pjbp7"
Oct 30 18:55:47.195233 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:47.195203    4924 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 18:55:47.195601 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:47.195253    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk" podUID="7d743094-1851-4c3a-9866-ecb1062c69b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 18:55:47.292333 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:55:47.292291    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a267a84_7dc4_4098_a7d3_4b0e8fb1867e.slice/crio-e6cc564b2bf9ad6f793e9dd078d36363e86b2ab2330d6ded1102bc5e10939449 WatchSource:0}: Error finding container e6cc564b2bf9ad6f793e9dd078d36363e86b2ab2330d6ded1102bc5e10939449: Status 404 returned error can't find the container with id e6cc564b2bf9ad6f793e9dd078d36363e86b2ab2330d6ded1102bc5e10939449
Oct 30 18:55:47.293021 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:47.292992    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fdecb3b-zpc44"]
Oct 30 18:55:47.396266 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:47.396211    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e280a1e-pjbp7"]
Oct 30 18:55:47.397059 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:55:47.397035    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda545e6e7_3f1f_42ed_8f96_147bef8d70c6.slice/crio-8dc7fa72d113a64f3a2490cb38a0160e02fa54453ca1ce53db277619d3d501cf WatchSource:0}: Error finding container 8dc7fa72d113a64f3a2490cb38a0160e02fa54453ca1ce53db277619d3d501cf: Status 404 returned error can't find the container with id 8dc7fa72d113a64f3a2490cb38a0160e02fa54453ca1ce53db277619d3d501cf
Oct 30 18:55:47.599749 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:47.599713    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fdecb3b-zpc44" event={"ID":"9a267a84-7dc4-4098-a7d3-4b0e8fb1867e","Type":"ContainerStarted","Data":"e6cc564b2bf9ad6f793e9dd078d36363e86b2ab2330d6ded1102bc5e10939449"}
Oct 30 18:55:47.601133 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:47.601104    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e280a1e-pjbp7" event={"ID":"a545e6e7-3f1f-42ed-8f96-147bef8d70c6","Type":"ContainerStarted","Data":"8dc7fa72d113a64f3a2490cb38a0160e02fa54453ca1ce53db277619d3d501cf"}
Oct 30 18:55:47.717079 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:47.717024    4924 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 18:55:47.717244 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:47.717112    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn" podUID="816f124c-a4bb-478e-b48a-c8b91a4113d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 18:55:47.885252 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:47.885181    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"]
Oct 30 18:55:47.890277 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:47.890258    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:55:47.948097 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:47.942219    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a1617e\"/\"openshift-service-ca.crt\""
Oct 30 18:55:47.948097 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:47.945817    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"]
Oct 30 18:55:47.969290 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:47.969248    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-tmp-dir\") pod \"distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l\" (UID: \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:55:47.969474 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:47.969412    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4czhc\" (UniqueName: \"kubernetes.io/projected/bdc8df4b-8c94-40b8-b4be-80d1432d87de-kube-api-access-4czhc\") pod \"distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l\" (UID: \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:55:47.969548 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:47.969467    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdc8df4b-8c94-40b8-b4be-80d1432d87de-tls-certs\") pod \"distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l\" (UID: \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:55:47.969548 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:47.969519    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-model-cache\") pod \"distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l\" (UID: \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:55:47.969673 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:47.969655    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-home\") pod \"distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l\" (UID: \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:55:47.969716 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:47.969693    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-dshm\") pod \"distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l\" (UID: \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:55:47.984481 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:47.984456    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a1617e\"/\"kube-root-ca.crt\""
Oct 30 18:55:47.984637 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:47.984622    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a1617e\"/\"default-dockercfg-xbgmq\""
Oct 30 18:55:47.984706 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:47.984661    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a1617e\"/\"distinct-w3-a1617e-kserve-self-signed-certs\""
Oct 30 18:55:48.033548 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.033520    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:55:48.073089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.070640    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-home\") pod \"distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l\" (UID: \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:55:48.073089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.070689    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-dshm\") pod \"distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l\" (UID: \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:55:48.073089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.070818    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-tmp-dir\") pod \"distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l\" (UID: \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:55:48.073089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.070886    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4czhc\" (UniqueName: \"kubernetes.io/projected/bdc8df4b-8c94-40b8-b4be-80d1432d87de-kube-api-access-4czhc\") pod \"distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l\" (UID: \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:55:48.073089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.070925    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdc8df4b-8c94-40b8-b4be-80d1432d87de-tls-certs\") pod \"distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l\" (UID: \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:55:48.073089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.070961    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-model-cache\") pod \"distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l\" (UID: \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:55:48.073089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.071429    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-model-cache\") pod \"distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l\" (UID: \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:55:48.073089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.071648    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-home\") pod \"distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l\" (UID: \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:55:48.073089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.072738    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-tmp-dir\") pod \"distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l\" (UID: \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:55:48.074189 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.074163    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-dshm\") pod \"distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l\" (UID: \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:55:48.075525 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.075301    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdc8df4b-8c94-40b8-b4be-80d1432d87de-tls-certs\") pod \"distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l\" (UID: \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:55:48.127817 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.127790    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4czhc\" (UniqueName: \"kubernetes.io/projected/bdc8df4b-8c94-40b8-b4be-80d1432d87de-kube-api-access-4czhc\") pod \"distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l\" (UID: \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:55:48.198722 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.198644    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:48.199119 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.199033    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:48.199729 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.199706    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 18:55:48.204941 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.204918    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:55:48.577903 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.577872    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"]
Oct 30 18:55:48.579597 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:55:48.579561    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdc8df4b_8c94_40b8_b4be_80d1432d87de.slice/crio-b70973ac682e2ac4c555b24b7a01855049f7ccf4612cbe6b9d37a50d2c26872a WatchSource:0}: Error finding container b70973ac682e2ac4c555b24b7a01855049f7ccf4612cbe6b9d37a50d2c26872a: Status 404 returned error can't find the container with id b70973ac682e2ac4c555b24b7a01855049f7ccf4612cbe6b9d37a50d2c26872a
Oct 30 18:55:48.609981 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.609956    4924 generic.go:356] "Generic (PLEG): container finished" podID="9a267a84-7dc4-4098-a7d3-4b0e8fb1867e" containerID="b149188310ded77d35a9afb536152b0fde44529ca9629e640ccfb823f3f5a215" exitCode=0
Oct 30 18:55:48.610101 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.610036    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fdecb3b-zpc44" event={"ID":"9a267a84-7dc4-4098-a7d3-4b0e8fb1867e","Type":"ContainerDied","Data":"b149188310ded77d35a9afb536152b0fde44529ca9629e640ccfb823f3f5a215"}
Oct 30 18:55:48.611617 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.611595    4924 generic.go:356] "Generic (PLEG): container finished" podID="a545e6e7-3f1f-42ed-8f96-147bef8d70c6" containerID="ac0cb6591f40e0094f978150d417ffc12e4acc5b9e9042a0e917e7ddfd2852f2" exitCode=0
Oct 30 18:55:48.611700 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.611670    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e280a1e-pjbp7" event={"ID":"a545e6e7-3f1f-42ed-8f96-147bef8d70c6","Type":"ContainerDied","Data":"ac0cb6591f40e0094f978150d417ffc12e4acc5b9e9042a0e917e7ddfd2852f2"}
Oct 30 18:55:48.613029 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.613011    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l" event={"ID":"bdc8df4b-8c94-40b8-b4be-80d1432d87de","Type":"ContainerStarted","Data":"b70973ac682e2ac4c555b24b7a01855049f7ccf4612cbe6b9d37a50d2c26872a"}
Oct 30 18:55:48.721631 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.721598    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:48.721984 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.721962    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:48.722841 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.722812    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 18:55:48.917228 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:48.917190    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:55:49.621132 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:49.621096    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l" event={"ID":"bdc8df4b-8c94-40b8-b4be-80d1432d87de","Type":"ContainerStarted","Data":"1f766806088d6f7a30f731ecb880a0a63113e8f4b3b7e11b044b92f5e4a399d8"}
Oct 30 18:55:49.731014 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:49.730965    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l" podStartSLOduration=2.730949738 podStartE2EDuration="2.730949738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:49.728105389 +0000 UTC m=+1151.005155833" watchObservedRunningTime="2026-10-30 18:55:49.730949738 +0000 UTC m=+1151.008000183"
Oct 30 18:55:49.791949 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:49.791923    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fdecb3b-zpc44"
Oct 30 18:55:49.819705 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:49.819687    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e280a1e-pjbp7"
Oct 30 18:55:49.894120 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:49.893999    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a267a84-7dc4-4098-a7d3-4b0e8fb1867e-kube-api-access-4snt7\" (UniqueName: \"kubernetes.io/projected/9a267a84-7dc4-4098-a7d3-4b0e8fb1867e-kube-api-access-4snt7\") pod \"9a267a84-7dc4-4098-a7d3-4b0e8fb1867e\" (UID: \"9a267a84-7dc4-4098-a7d3-4b0e8fb1867e\") "
Oct 30 18:55:49.894120 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:49.894036    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a545e6e7-3f1f-42ed-8f96-147bef8d70c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a545e6e7-3f1f-42ed-8f96-147bef8d70c6-maas-api-openshift-ca\") pod \"a545e6e7-3f1f-42ed-8f96-147bef8d70c6\" (UID: \"a545e6e7-3f1f-42ed-8f96-147bef8d70c6\") "
Oct 30 18:55:49.894120 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:49.894059    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a545e6e7-3f1f-42ed-8f96-147bef8d70c6-kube-api-access-6lf54\" (UniqueName: \"kubernetes.io/projected/a545e6e7-3f1f-42ed-8f96-147bef8d70c6-kube-api-access-6lf54\") pod \"a545e6e7-3f1f-42ed-8f96-147bef8d70c6\" (UID: \"a545e6e7-3f1f-42ed-8f96-147bef8d70c6\") "
Oct 30 18:55:49.894414 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:49.894180    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a545e6e7-3f1f-42ed-8f96-147bef8d70c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a545e6e7-3f1f-42ed-8f96-147bef8d70c6-maas-api-xks-ca\") pod \"a545e6e7-3f1f-42ed-8f96-147bef8d70c6\" (UID: \"a545e6e7-3f1f-42ed-8f96-147bef8d70c6\") "
Oct 30 18:55:49.894414 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:49.894265    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a267a84-7dc4-4098-a7d3-4b0e8fb1867e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a267a84-7dc4-4098-a7d3-4b0e8fb1867e-maas-api-xks-ca\") pod \"9a267a84-7dc4-4098-a7d3-4b0e8fb1867e\" (UID: \"9a267a84-7dc4-4098-a7d3-4b0e8fb1867e\") "
Oct 30 18:55:49.894515 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:49.894422    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a545e6e7-3f1f-42ed-8f96-147bef8d70c6-maas-api-openshift-ca" pod "a545e6e7-3f1f-42ed-8f96-147bef8d70c6" (UID: "a545e6e7-3f1f-42ed-8f96-147bef8d70c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:49.894515 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:49.894428    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a267a84-7dc4-4098-a7d3-4b0e8fb1867e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a267a84-7dc4-4098-a7d3-4b0e8fb1867e-maas-api-openshift-ca\") pod \"9a267a84-7dc4-4098-a7d3-4b0e8fb1867e\" (UID: \"9a267a84-7dc4-4098-a7d3-4b0e8fb1867e\") "
Oct 30 18:55:49.895465 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:49.894785    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a267a84-7dc4-4098-a7d3-4b0e8fb1867e-maas-api-openshift-ca" pod "9a267a84-7dc4-4098-a7d3-4b0e8fb1867e" (UID: "9a267a84-7dc4-4098-a7d3-4b0e8fb1867e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:55:49.895465 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:49.895077    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a267a84-7dc4-4098-a7d3-4b0e8fb1867e-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:49.895465 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:49.895102    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a545e6e7-3f1f-42ed-8f96-147bef8d70c6-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:49.896537 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:49.896503    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a267a84-7dc4-4098-a7d3-4b0e8fb1867e-kube-api-access-4snt7" pod "9a267a84-7dc4-4098-a7d3-4b0e8fb1867e" (UID: "9a267a84-7dc4-4098-a7d3-4b0e8fb1867e"). InnerVolumeSpecName "kube-api-access-4snt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:49.896778 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:49.896755    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a545e6e7-3f1f-42ed-8f96-147bef8d70c6-maas-api-xks-ca" pod "a545e6e7-3f1f-42ed-8f96-147bef8d70c6" (UID: "a545e6e7-3f1f-42ed-8f96-147bef8d70c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:49.896908 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:49.896890    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a267a84-7dc4-4098-a7d3-4b0e8fb1867e-maas-api-xks-ca" pod "9a267a84-7dc4-4098-a7d3-4b0e8fb1867e" (UID: "9a267a84-7dc4-4098-a7d3-4b0e8fb1867e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:55:49.896975 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:49.896952    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a545e6e7-3f1f-42ed-8f96-147bef8d70c6-kube-api-access-6lf54" pod "a545e6e7-3f1f-42ed-8f96-147bef8d70c6" (UID: "a545e6e7-3f1f-42ed-8f96-147bef8d70c6"). InnerVolumeSpecName "kube-api-access-6lf54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:55:49.996242 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:49.996207    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a545e6e7-3f1f-42ed-8f96-147bef8d70c6-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:49.996242 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:49.996235    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a267a84-7dc4-4098-a7d3-4b0e8fb1867e-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:49.996242 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:49.996245    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4snt7\" (UniqueName: \"kubernetes.io/projected/9a267a84-7dc4-4098-a7d3-4b0e8fb1867e-kube-api-access-4snt7\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:49.996540 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:49.996260    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lf54\" (UniqueName: \"kubernetes.io/projected/a545e6e7-3f1f-42ed-8f96-147bef8d70c6-kube-api-access-6lf54\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:55:50.205417 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:50.205336    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:55:50.629298 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:50.629269    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fdecb3b-zpc44"
Oct 30 18:55:50.629701 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:50.629268    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fdecb3b-zpc44" event={"ID":"9a267a84-7dc4-4098-a7d3-4b0e8fb1867e","Type":"ContainerDied","Data":"e6cc564b2bf9ad6f793e9dd078d36363e86b2ab2330d6ded1102bc5e10939449"}
Oct 30 18:55:50.629701 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:50.629395    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6cc564b2bf9ad6f793e9dd078d36363e86b2ab2330d6ded1102bc5e10939449"
Oct 30 18:55:50.630709 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:50.630681    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e280a1e-pjbp7"
Oct 30 18:55:50.630824 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:50.630724    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e280a1e-pjbp7" event={"ID":"a545e6e7-3f1f-42ed-8f96-147bef8d70c6","Type":"ContainerDied","Data":"8dc7fa72d113a64f3a2490cb38a0160e02fa54453ca1ce53db277619d3d501cf"}
Oct 30 18:55:50.630824 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:50.630766    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dc7fa72d113a64f3a2490cb38a0160e02fa54453ca1ce53db277619d3d501cf"
Oct 30 18:55:53.761703 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:53.761660    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:55:56.076910 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:56.076874    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:55:57.181618 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:57.181581    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af4a17ad-z2fqm"]
Oct 30 18:55:57.182379 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:57.182357    4924 cpu_manager.go:414] "containerMap: removing container" podUID="9a267a84-7dc4-4098-a7d3-4b0e8fb1867e" containerName="revoke-keys"
Oct 30 18:55:57.182379 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:57.182378    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a267a84-7dc4-4098-a7d3-4b0e8fb1867e" containerName="revoke-keys"
Oct 30 18:55:57.182570 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:57.182418    4924 cpu_manager.go:414] "containerMap: removing container" podUID="a545e6e7-3f1f-42ed-8f96-147bef8d70c6" containerName="revoke-keys"
Oct 30 18:55:57.182570 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:57.182426    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a545e6e7-3f1f-42ed-8f96-147bef8d70c6" containerName="revoke-keys"
Oct 30 18:55:57.182673 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:57.182597    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a267a84-7dc4-4098-a7d3-4b0e8fb1867e" containerName="revoke-keys"
Oct 30 18:55:57.182673 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:57.182616    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="a545e6e7-3f1f-42ed-8f96-147bef8d70c6" containerName="revoke-keys"
Oct 30 18:55:57.186224 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:57.186203    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af4a17ad-z2fqm"
Oct 30 18:55:57.240211 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:57.240153    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af4a17ad-z2fqm"]
Oct 30 18:55:57.281891 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:57.281859    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:55:57.282046 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:57.281954    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 18:55:57.282046 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:57.282001    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:55:57.282166 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:57.282153    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:55:57.376329 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:57.376292    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0db98356-328c-4cb0-8641-2d7d88f38d96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af4a17ad-z2fqm\" (UID: \"0db98356-328c-4cb0-8641-2d7d88f38d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af4a17ad-z2fqm"
Oct 30 18:55:57.376489 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:57.376361    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtjxp\" (UniqueName: \"kubernetes.io/projected/0db98356-328c-4cb0-8641-2d7d88f38d96-kube-api-access-jtjxp\") pod \"maas-api-revoke-sub-af4a17ad-z2fqm\" (UID: \"0db98356-328c-4cb0-8641-2d7d88f38d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af4a17ad-z2fqm"
Oct 30 18:55:57.376489 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:57.376458    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0db98356-328c-4cb0-8641-2d7d88f38d96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af4a17ad-z2fqm\" (UID: \"0db98356-328c-4cb0-8641-2d7d88f38d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af4a17ad-z2fqm"
Oct 30 18:55:57.477747 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:57.477647    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtjxp\" (UniqueName: \"kubernetes.io/projected/0db98356-328c-4cb0-8641-2d7d88f38d96-kube-api-access-jtjxp\") pod \"maas-api-revoke-sub-af4a17ad-z2fqm\" (UID: \"0db98356-328c-4cb0-8641-2d7d88f38d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af4a17ad-z2fqm"
Oct 30 18:55:57.477747 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:57.477712    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0db98356-328c-4cb0-8641-2d7d88f38d96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af4a17ad-z2fqm\" (UID: \"0db98356-328c-4cb0-8641-2d7d88f38d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af4a17ad-z2fqm"
Oct 30 18:55:57.477995 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:57.477943    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0db98356-328c-4cb0-8641-2d7d88f38d96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af4a17ad-z2fqm\" (UID: \"0db98356-328c-4cb0-8641-2d7d88f38d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af4a17ad-z2fqm"
Oct 30 18:55:57.478919 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:57.478892    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0db98356-328c-4cb0-8641-2d7d88f38d96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af4a17ad-z2fqm\" (UID: \"0db98356-328c-4cb0-8641-2d7d88f38d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af4a17ad-z2fqm"
Oct 30 18:55:57.482487 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:57.482461    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0db98356-328c-4cb0-8641-2d7d88f38d96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af4a17ad-z2fqm\" (UID: \"0db98356-328c-4cb0-8641-2d7d88f38d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af4a17ad-z2fqm"
Oct 30 18:55:57.534216 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:57.534178    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtjxp\" (UniqueName: \"kubernetes.io/projected/0db98356-328c-4cb0-8641-2d7d88f38d96-kube-api-access-jtjxp\") pod \"maas-api-revoke-sub-af4a17ad-z2fqm\" (UID: \"0db98356-328c-4cb0-8641-2d7d88f38d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af4a17ad-z2fqm"
Oct 30 18:55:57.800630 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:57.800533    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af4a17ad-z2fqm"
Oct 30 18:55:58.205420 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:58.205380    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:55:58.216267 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:58.215041    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af4a17ad-z2fqm"]
Oct 30 18:55:58.218002 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:55:58.217978    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db98356_328c_4cb0_8641_2d7d88f38d96.slice/crio-7b043d0a6fb26a17a5948b7421ea8d035e10b43ff33a1273fae69cc4eec22e90 WatchSource:0}: Error finding container 7b043d0a6fb26a17a5948b7421ea8d035e10b43ff33a1273fae69cc4eec22e90: Status 404 returned error can't find the container with id 7b043d0a6fb26a17a5948b7421ea8d035e10b43ff33a1273fae69cc4eec22e90
Oct 30 18:55:58.224187 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:58.224164    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:55:58.523482 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:58.523444    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:55:58.696195 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:58.696153    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af4a17ad-z2fqm" event={"ID":"0db98356-328c-4cb0-8641-2d7d88f38d96","Type":"ContainerStarted","Data":"990c33e4581f4d154128a37a69aa82f23e018114a49444cc2e519846f90db190"}
Oct 30 18:55:58.696397 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:58.696246    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af4a17ad-z2fqm" event={"ID":"0db98356-328c-4cb0-8641-2d7d88f38d96","Type":"ContainerStarted","Data":"7b043d0a6fb26a17a5948b7421ea8d035e10b43ff33a1273fae69cc4eec22e90"}
Oct 30 18:55:58.708227 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:58.708204    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:55:58.810403 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:58.810297    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af4a17ad-z2fqm" podStartSLOduration=1.81027947 podStartE2EDuration="1.81027947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:58.809164367 +0000 UTC m=+1160.086214811" watchObservedRunningTime="2026-10-30 18:55:58.81027947 +0000 UTC m=+1160.087329917"
Oct 30 18:55:59.705007 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:59.704972    4924 generic.go:356] "Generic (PLEG): container finished" podID="0db98356-328c-4cb0-8641-2d7d88f38d96" containerID="990c33e4581f4d154128a37a69aa82f23e018114a49444cc2e519846f90db190" exitCode=0
Oct 30 18:55:59.705472 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:55:59.705053    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af4a17ad-z2fqm" event={"ID":"0db98356-328c-4cb0-8641-2d7d88f38d96","Type":"ContainerDied","Data":"990c33e4581f4d154128a37a69aa82f23e018114a49444cc2e519846f90db190"}
Oct 30 18:56:00.863831 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:00.863805    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af4a17ad-z2fqm"
Oct 30 18:56:00.918010 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:00.917982    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0db98356-328c-4cb0-8641-2d7d88f38d96-kube-api-access-jtjxp\" (UniqueName: \"kubernetes.io/projected/0db98356-328c-4cb0-8641-2d7d88f38d96-kube-api-access-jtjxp\") pod \"0db98356-328c-4cb0-8641-2d7d88f38d96\" (UID: \"0db98356-328c-4cb0-8641-2d7d88f38d96\") "
Oct 30 18:56:00.918212 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:00.918198    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0db98356-328c-4cb0-8641-2d7d88f38d96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0db98356-328c-4cb0-8641-2d7d88f38d96-maas-api-openshift-ca\") pod \"0db98356-328c-4cb0-8641-2d7d88f38d96\" (UID: \"0db98356-328c-4cb0-8641-2d7d88f38d96\") "
Oct 30 18:56:00.918294 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:00.918228    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0db98356-328c-4cb0-8641-2d7d88f38d96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0db98356-328c-4cb0-8641-2d7d88f38d96-maas-api-xks-ca\") pod \"0db98356-328c-4cb0-8641-2d7d88f38d96\" (UID: \"0db98356-328c-4cb0-8641-2d7d88f38d96\") "
Oct 30 18:56:00.918532 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:00.918510    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db98356-328c-4cb0-8641-2d7d88f38d96-maas-api-openshift-ca" pod "0db98356-328c-4cb0-8641-2d7d88f38d96" (UID: "0db98356-328c-4cb0-8641-2d7d88f38d96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:00.918654 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:00.918636    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0db98356-328c-4cb0-8641-2d7d88f38d96-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:00.920124 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:00.920099    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db98356-328c-4cb0-8641-2d7d88f38d96-kube-api-access-jtjxp" pod "0db98356-328c-4cb0-8641-2d7d88f38d96" (UID: "0db98356-328c-4cb0-8641-2d7d88f38d96"). InnerVolumeSpecName "kube-api-access-jtjxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:00.920252 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:00.920234    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0db98356-328c-4cb0-8641-2d7d88f38d96-maas-api-xks-ca" pod "0db98356-328c-4cb0-8641-2d7d88f38d96" (UID: "0db98356-328c-4cb0-8641-2d7d88f38d96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:01.019465 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:01.019398    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0db98356-328c-4cb0-8641-2d7d88f38d96-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:01.019465 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:01.019423    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtjxp\" (UniqueName: \"kubernetes.io/projected/0db98356-328c-4cb0-8641-2d7d88f38d96-kube-api-access-jtjxp\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:01.719766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:01.719735    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af4a17ad-z2fqm"
Oct 30 18:56:01.719948 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:01.719779    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af4a17ad-z2fqm" event={"ID":"0db98356-328c-4cb0-8641-2d7d88f38d96","Type":"ContainerDied","Data":"7b043d0a6fb26a17a5948b7421ea8d035e10b43ff33a1273fae69cc4eec22e90"}
Oct 30 18:56:01.719948 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:01.719825    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b043d0a6fb26a17a5948b7421ea8d035e10b43ff33a1273fae69cc4eec22e90"
Oct 30 18:56:01.860276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:01.860228    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a4f200e-7gb7p"]
Oct 30 18:56:01.861221 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:01.861194    4924 cpu_manager.go:414] "containerMap: removing container" podUID="0db98356-328c-4cb0-8641-2d7d88f38d96" containerName="revoke-keys"
Oct 30 18:56:01.861221 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:01.861221    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0db98356-328c-4cb0-8641-2d7d88f38d96" containerName="revoke-keys"
Oct 30 18:56:01.861415 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:01.861403    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="0db98356-328c-4cb0-8641-2d7d88f38d96" containerName="revoke-keys"
Oct 30 18:56:01.865655 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:01.865633    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4f200e-7gb7p"
Oct 30 18:56:01.915527 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:01.915490    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:56:01.915709 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:01.915593    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:56:01.915832 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:01.915808    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a4f200e-7gb7p"]
Oct 30 18:56:01.929543 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:01.929504    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d4c6b1b-e52c-4b98-bf41-b8de7a282623-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a4f200e-7gb7p\" (UID: \"3d4c6b1b-e52c-4b98-bf41-b8de7a282623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4f200e-7gb7p"
Oct 30 18:56:01.929720 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:01.929681    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6kp9\" (UniqueName: \"kubernetes.io/projected/3d4c6b1b-e52c-4b98-bf41-b8de7a282623-kube-api-access-h6kp9\") pod \"maas-api-revoke-sub-8a4f200e-7gb7p\" (UID: \"3d4c6b1b-e52c-4b98-bf41-b8de7a282623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4f200e-7gb7p"
Oct 30 18:56:01.929774 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:01.929723    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d4c6b1b-e52c-4b98-bf41-b8de7a282623-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a4f200e-7gb7p\" (UID: \"3d4c6b1b-e52c-4b98-bf41-b8de7a282623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4f200e-7gb7p"
Oct 30 18:56:01.957460 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:01.957428    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:56:01.957631 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:01.957467    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 18:56:02.031391 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:02.031293    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d4c6b1b-e52c-4b98-bf41-b8de7a282623-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a4f200e-7gb7p\" (UID: \"3d4c6b1b-e52c-4b98-bf41-b8de7a282623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4f200e-7gb7p"
Oct 30 18:56:02.031391 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:02.031385    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6kp9\" (UniqueName: \"kubernetes.io/projected/3d4c6b1b-e52c-4b98-bf41-b8de7a282623-kube-api-access-h6kp9\") pod \"maas-api-revoke-sub-8a4f200e-7gb7p\" (UID: \"3d4c6b1b-e52c-4b98-bf41-b8de7a282623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4f200e-7gb7p"
Oct 30 18:56:02.031612 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:02.031420    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d4c6b1b-e52c-4b98-bf41-b8de7a282623-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a4f200e-7gb7p\" (UID: \"3d4c6b1b-e52c-4b98-bf41-b8de7a282623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4f200e-7gb7p"
Oct 30 18:56:02.032136 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:02.032113    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d4c6b1b-e52c-4b98-bf41-b8de7a282623-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a4f200e-7gb7p\" (UID: \"3d4c6b1b-e52c-4b98-bf41-b8de7a282623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4f200e-7gb7p"
Oct 30 18:56:02.033666 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:02.033644    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d4c6b1b-e52c-4b98-bf41-b8de7a282623-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a4f200e-7gb7p\" (UID: \"3d4c6b1b-e52c-4b98-bf41-b8de7a282623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4f200e-7gb7p"
Oct 30 18:56:02.085929 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:02.085897    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6kp9\" (UniqueName: \"kubernetes.io/projected/3d4c6b1b-e52c-4b98-bf41-b8de7a282623-kube-api-access-h6kp9\") pod \"maas-api-revoke-sub-8a4f200e-7gb7p\" (UID: \"3d4c6b1b-e52c-4b98-bf41-b8de7a282623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4f200e-7gb7p"
Oct 30 18:56:02.181343 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:02.181309    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4f200e-7gb7p"
Oct 30 18:56:02.550943 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:56:02.550236    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d4c6b1b_e52c_4b98_bf41_b8de7a282623.slice/crio-c60ecf6707f4f91a85109578d79d99782eccffe2f3662d3e25259dc9f4d3a281 WatchSource:0}: Error finding container c60ecf6707f4f91a85109578d79d99782eccffe2f3662d3e25259dc9f4d3a281: Status 404 returned error can't find the container with id c60ecf6707f4f91a85109578d79d99782eccffe2f3662d3e25259dc9f4d3a281
Oct 30 18:56:02.550943 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:02.550825    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a4f200e-7gb7p"]
Oct 30 18:56:02.729862 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:02.729829    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4f200e-7gb7p" event={"ID":"3d4c6b1b-e52c-4b98-bf41-b8de7a282623","Type":"ContainerStarted","Data":"c60ecf6707f4f91a85109578d79d99782eccffe2f3662d3e25259dc9f4d3a281"}
Oct 30 18:56:03.319713 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:03.319659    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:03.737674 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:03.737641    4924 generic.go:356] "Generic (PLEG): container finished" podID="3d4c6b1b-e52c-4b98-bf41-b8de7a282623" containerID="77cef0164c1776a2ab88562814cbea8540494084c4cb68f28d9ddaa33f5ffd75" exitCode=0
Oct 30 18:56:03.737872 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:03.737726    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4f200e-7gb7p" event={"ID":"3d4c6b1b-e52c-4b98-bf41-b8de7a282623","Type":"ContainerDied","Data":"77cef0164c1776a2ab88562814cbea8540494084c4cb68f28d9ddaa33f5ffd75"}
Oct 30 18:56:04.908871 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:04.908846    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4f200e-7gb7p"
Oct 30 18:56:04.967668 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:04.967635    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d4c6b1b-e52c-4b98-bf41-b8de7a282623-kube-api-access-h6kp9\" (UniqueName: \"kubernetes.io/projected/3d4c6b1b-e52c-4b98-bf41-b8de7a282623-kube-api-access-h6kp9\") pod \"3d4c6b1b-e52c-4b98-bf41-b8de7a282623\" (UID: \"3d4c6b1b-e52c-4b98-bf41-b8de7a282623\") "
Oct 30 18:56:04.967835 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:04.967680    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d4c6b1b-e52c-4b98-bf41-b8de7a282623-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d4c6b1b-e52c-4b98-bf41-b8de7a282623-maas-api-xks-ca\") pod \"3d4c6b1b-e52c-4b98-bf41-b8de7a282623\" (UID: \"3d4c6b1b-e52c-4b98-bf41-b8de7a282623\") "
Oct 30 18:56:04.967835 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:04.967725    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d4c6b1b-e52c-4b98-bf41-b8de7a282623-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d4c6b1b-e52c-4b98-bf41-b8de7a282623-maas-api-openshift-ca\") pod \"3d4c6b1b-e52c-4b98-bf41-b8de7a282623\" (UID: \"3d4c6b1b-e52c-4b98-bf41-b8de7a282623\") "
Oct 30 18:56:04.968129 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:04.968101    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d4c6b1b-e52c-4b98-bf41-b8de7a282623-maas-api-openshift-ca" pod "3d4c6b1b-e52c-4b98-bf41-b8de7a282623" (UID: "3d4c6b1b-e52c-4b98-bf41-b8de7a282623"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:56:04.968257 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:04.968243    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d4c6b1b-e52c-4b98-bf41-b8de7a282623-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:04.969812 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:04.969786    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d4c6b1b-e52c-4b98-bf41-b8de7a282623-maas-api-xks-ca" pod "3d4c6b1b-e52c-4b98-bf41-b8de7a282623" (UID: "3d4c6b1b-e52c-4b98-bf41-b8de7a282623"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:56:04.969895 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:04.969852    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d4c6b1b-e52c-4b98-bf41-b8de7a282623-kube-api-access-h6kp9" pod "3d4c6b1b-e52c-4b98-bf41-b8de7a282623" (UID: "3d4c6b1b-e52c-4b98-bf41-b8de7a282623"). InnerVolumeSpecName "kube-api-access-h6kp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:56:05.069590 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:05.069512    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6kp9\" (UniqueName: \"kubernetes.io/projected/3d4c6b1b-e52c-4b98-bf41-b8de7a282623-kube-api-access-h6kp9\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:05.069590 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:05.069540    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d4c6b1b-e52c-4b98-bf41-b8de7a282623-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:56:05.752990 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:05.752954    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4f200e-7gb7p"
Oct 30 18:56:05.753188 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:05.752984    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a4f200e-7gb7p" event={"ID":"3d4c6b1b-e52c-4b98-bf41-b8de7a282623","Type":"ContainerDied","Data":"c60ecf6707f4f91a85109578d79d99782eccffe2f3662d3e25259dc9f4d3a281"}
Oct 30 18:56:05.753188 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:05.753038    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c60ecf6707f4f91a85109578d79d99782eccffe2f3662d3e25259dc9f4d3a281"
Oct 30 18:56:08.048562 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:08.048464    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:12.923798 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:12.923754    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:14.664805 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.664768    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"]
Oct 30 18:56:14.665374 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.665360    4924 cpu_manager.go:414] "containerMap: removing container" podUID="3d4c6b1b-e52c-4b98-bf41-b8de7a282623" containerName="revoke-keys"
Oct 30 18:56:14.665423 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.665375    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d4c6b1b-e52c-4b98-bf41-b8de7a282623" containerName="revoke-keys"
Oct 30 18:56:14.665473 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.665464    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d4c6b1b-e52c-4b98-bf41-b8de7a282623" containerName="revoke-keys"
Oct 30 18:56:14.670506 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.670481    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:56:14.725293 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.725261    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"]
Oct 30 18:56:14.766608 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.766575    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f5705f\"/\"distinct-w0-f5705f-kserve-self-signed-certs\""
Oct 30 18:56:14.772136 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.772107    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-dshm\") pod \"distinct-w0-f5705f-kserve-5585ddc876-4blrg\" (UID: \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\") " pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:56:14.772248 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.772166    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-tmp-dir\") pod \"distinct-w0-f5705f-kserve-5585ddc876-4blrg\" (UID: \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\") " pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:56:14.772321 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.772299    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjx2h\" (UniqueName: \"kubernetes.io/projected/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-kube-api-access-pjx2h\") pod \"distinct-w0-f5705f-kserve-5585ddc876-4blrg\" (UID: \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\") " pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:56:14.772369 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.772363    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-home\") pod \"distinct-w0-f5705f-kserve-5585ddc876-4blrg\" (UID: \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\") " pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:56:14.772473 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.772457    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-model-cache\") pod \"distinct-w0-f5705f-kserve-5585ddc876-4blrg\" (UID: \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\") " pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:56:14.772573 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.772560    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-tls-certs\") pod \"distinct-w0-f5705f-kserve-5585ddc876-4blrg\" (UID: \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\") " pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:56:14.812398 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.812364    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:14.888088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.877179    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-model-cache\") pod \"distinct-w0-f5705f-kserve-5585ddc876-4blrg\" (UID: \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\") " pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:56:14.888088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.877315    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-tls-certs\") pod \"distinct-w0-f5705f-kserve-5585ddc876-4blrg\" (UID: \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\") " pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:56:14.888088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.877370    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-dshm\") pod \"distinct-w0-f5705f-kserve-5585ddc876-4blrg\" (UID: \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\") " pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:56:14.888088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.877428    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-tmp-dir\") pod \"distinct-w0-f5705f-kserve-5585ddc876-4blrg\" (UID: \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\") " pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:56:14.888088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.877570    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjx2h\" (UniqueName: \"kubernetes.io/projected/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-kube-api-access-pjx2h\") pod \"distinct-w0-f5705f-kserve-5585ddc876-4blrg\" (UID: \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\") " pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:56:14.888088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.877783    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-home\") pod \"distinct-w0-f5705f-kserve-5585ddc876-4blrg\" (UID: \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\") " pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:56:14.896651 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.896614    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-model-cache\") pod \"distinct-w0-f5705f-kserve-5585ddc876-4blrg\" (UID: \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\") " pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:56:14.897617 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.897596    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-tmp-dir\") pod \"distinct-w0-f5705f-kserve-5585ddc876-4blrg\" (UID: \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\") " pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:56:14.898455 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.898435    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-home\") pod \"distinct-w0-f5705f-kserve-5585ddc876-4blrg\" (UID: \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\") " pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:56:14.902382 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.902356    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-dshm\") pod \"distinct-w0-f5705f-kserve-5585ddc876-4blrg\" (UID: \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\") " pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:56:14.912105 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.912059    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-tls-certs\") pod \"distinct-w0-f5705f-kserve-5585ddc876-4blrg\" (UID: \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\") " pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:56:14.952801 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.952718    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjx2h\" (UniqueName: \"kubernetes.io/projected/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-kube-api-access-pjx2h\") pod \"distinct-w0-f5705f-kserve-5585ddc876-4blrg\" (UID: \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\") " pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:56:14.983461 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:14.983430    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:56:15.368725 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:15.368691    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"]
Oct 30 18:56:15.727142 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:56:15.725851    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:56:15.828245 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:15.828194    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg" event={"ID":"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e","Type":"ContainerStarted","Data":"b93aaa5f51acab31e5206a72c081fcfb623709688ad6cd5a0b8a6a7f1fb25c76"}
Oct 30 18:56:15.828245 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:15.828248    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg" event={"ID":"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e","Type":"ContainerStarted","Data":"812432fe32842d04cbf9b3f6c2024eb64eb9e95c4b6b9fbe988681db6f77f15d"}
Oct 30 18:56:15.944077 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:15.943991    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg" podStartSLOduration=1.943972487 podStartE2EDuration="1.943972487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:15.937363192 +0000 UTC m=+1177.214413637" watchObservedRunningTime="2026-10-30 18:56:15.943972487 +0000 UTC m=+1177.221022934"
Oct 30 18:56:15.983714 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:15.983624    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:56:17.708831 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:17.708795    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:22.494733 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:22.494696    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:24.984537 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:24.984509    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:56:24.996720 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:24.996695    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:56:25.907380 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:25.907345    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:56:27.286044 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:27.286005    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:32.124117 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:32.124056    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:34.859126 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:34.859088    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:36.664882 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:36.664844    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"]
Oct 30 18:56:36.668734 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:36.668715    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:56:36.729808 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:36.729771    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxwmz\" (UniqueName: \"kubernetes.io/projected/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-kube-api-access-zxwmz\") pod \"distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj\" (UID: \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:56:36.729973 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:36.729939    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-model-cache\") pod \"distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj\" (UID: \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:56:36.730032 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:36.729978    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-home\") pod \"distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj\" (UID: \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:56:36.730032 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:36.730007    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-tls-certs\") pod \"distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj\" (UID: \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:56:36.730169 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:36.730154    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-dshm\") pod \"distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj\" (UID: \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:56:36.730214 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:36.730196    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-tmp-dir\") pod \"distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj\" (UID: \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:56:36.739899 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:36.739877    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"]
Oct 30 18:56:36.765709 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:36.765681    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ec8dab\"/\"distinct-2-w2-ec8dab-kserve-self-signed-certs\""
Oct 30 18:56:36.804132 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:36.804090    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:36.831348 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:36.831319    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-dshm\") pod \"distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj\" (UID: \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:56:36.831492 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:36.831367    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-tmp-dir\") pod \"distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj\" (UID: \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:56:36.831492 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:36.831416    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxwmz\" (UniqueName: \"kubernetes.io/projected/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-kube-api-access-zxwmz\") pod \"distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj\" (UID: \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:56:36.831607 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:36.831495    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-model-cache\") pod \"distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj\" (UID: \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:56:36.831607 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:36.831528    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-home\") pod \"distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj\" (UID: \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:56:36.831607 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:36.831563    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-tls-certs\") pod \"distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj\" (UID: \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:56:36.831831 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:36.831808    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-tmp-dir\") pod \"distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj\" (UID: \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:56:36.831889 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:36.831864    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-model-cache\") pod \"distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj\" (UID: \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:56:36.831944 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:36.831926    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-home\") pod \"distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj\" (UID: \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:56:36.833718 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:36.833693    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-dshm\") pod \"distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj\" (UID: \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:56:36.833996 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:36.833977    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-tls-certs\") pod \"distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj\" (UID: \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:56:36.886545 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:36.886515    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxwmz\" (UniqueName: \"kubernetes.io/projected/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-kube-api-access-zxwmz\") pod \"distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj\" (UID: \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\") " pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:56:36.981698 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:36.981621    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:56:36.985675 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:36.985651    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:37.347593 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:56:37.347565    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51f3e639_9cd4_4aaa_a568_f19b6d2704fd.slice/crio-a30e5ad2631335ba7633794cfe6eb7f8800c53e439aea833eb459a4d81c578af WatchSource:0}: Error finding container a30e5ad2631335ba7633794cfe6eb7f8800c53e439aea833eb459a4d81c578af: Status 404 returned error can't find the container with id a30e5ad2631335ba7633794cfe6eb7f8800c53e439aea833eb459a4d81c578af
Oct 30 18:56:37.348054 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:37.348034    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"]
Oct 30 18:56:37.984313 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:37.984271    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj" event={"ID":"51f3e639-9cd4-4aaa-a568-f19b6d2704fd","Type":"ContainerStarted","Data":"aa0f8e048bb1d418eac1c5eb0d71205e7830315a6f63f556312a1f9aa6c504fa"}
Oct 30 18:56:37.984313 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:37.984312    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj" event={"ID":"51f3e639-9cd4-4aaa-a568-f19b6d2704fd","Type":"ContainerStarted","Data":"a30e5ad2631335ba7633794cfe6eb7f8800c53e439aea833eb459a4d81c578af"}
Oct 30 18:56:38.093090 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:38.093030    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj" podStartSLOduration=2.093015953 podStartE2EDuration="2.093015953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:38.092011993 +0000 UTC m=+1199.369062438" watchObservedRunningTime="2026-10-30 18:56:38.093015953 +0000 UTC m=+1199.370066419"
Oct 30 18:56:38.981757 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:38.981725    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:56:38.993659 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:38.993631    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 18:56:38.997839 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:38.997815    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 18:56:40.268689 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.268647    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"]
Oct 30 18:56:40.273254 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.273233    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.370671 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.370401    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-envoy\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.370671 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.370453    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-workload-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.370671 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.370479    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/971fec4f-5a07-4eae-b318-0e9c8f381983-istiod-ca-cert\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.370671 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.370511    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-data\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.370671 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.370570    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-workload-certs\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.370671 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.370605    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qn7m\" (UniqueName: \"kubernetes.io/projected/971fec4f-5a07-4eae-b318-0e9c8f381983-kube-api-access-2qn7m\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.371622 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.371186    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-token\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.371622 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.371359    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-credential-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.371622 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.371410    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-podinfo\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.375805 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.375784    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"]
Oct 30 18:56:40.472712 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.472682    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-workload-certs\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.472881 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.472722    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qn7m\" (UniqueName: \"kubernetes.io/projected/971fec4f-5a07-4eae-b318-0e9c8f381983-kube-api-access-2qn7m\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.472881 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.472787    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-token\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.472952 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.472891    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-credential-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.472952 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.472940    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-podinfo\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.473033 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.472981    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-envoy\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.473033 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.473009    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-workload-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.473146 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.473035    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/971fec4f-5a07-4eae-b318-0e9c8f381983-istiod-ca-cert\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.473146 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.473088    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-workload-certs\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.473254 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.473159    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-data\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.473333 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.473309    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-credential-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.473510 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.473484    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-workload-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.473623 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.473494    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-data\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.473748 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.473729    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/971fec4f-5a07-4eae-b318-0e9c8f381983-istiod-ca-cert\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.475250 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.475232    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-envoy\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.475394 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.475380    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-podinfo\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.526446 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.526386    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-token\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.526636 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.526619    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qn7m\" (UniqueName: \"kubernetes.io/projected/971fec4f-5a07-4eae-b318-0e9c8f381983-kube-api-access-2qn7m\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:40.587419 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:40.587394    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:41.003442 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:41.003412    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"]
Oct 30 18:56:41.004214 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:56:41.004177    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971fec4f_5a07_4eae_b318_0e9c8f381983.slice/crio-f15eb0ee74d8f5fca6faa0f4c6110c0ebb6dd36991a6bb684ede468bc65266b6 WatchSource:0}: Error finding container f15eb0ee74d8f5fca6faa0f4c6110c0ebb6dd36991a6bb684ede468bc65266b6: Status 404 returned error can't find the container with id f15eb0ee74d8f5fca6faa0f4c6110c0ebb6dd36991a6bb684ede468bc65266b6
Oct 30 18:56:41.006677 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:41.006645    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:56:41.006785 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:41.006720    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:56:41.006785 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:41.006766    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:56:41.819620 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:41.819579    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:42.040132 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:42.040092    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh" event={"ID":"971fec4f-5a07-4eae-b318-0e9c8f381983","Type":"ContainerStarted","Data":"57e1d57c7966a5575a5dfe2d548d6e30d4b99a905147c2f995c80e275c47bb63"}
Oct 30 18:56:42.040316 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:42.040191    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh" event={"ID":"971fec4f-5a07-4eae-b318-0e9c8f381983","Type":"ContainerStarted","Data":"f15eb0ee74d8f5fca6faa0f4c6110c0ebb6dd36991a6bb684ede468bc65266b6"}
Oct 30 18:56:42.244946 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:42.244884    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh" podStartSLOduration=2.244867202 podStartE2EDuration="2.244867202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:42.196480254 +0000 UTC m=+1203.473530699" watchObservedRunningTime="2026-10-30 18:56:42.244867202 +0000 UTC m=+1203.521917648"
Oct 30 18:56:42.588136 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:42.588033    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:42.589724 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:42.589690    4924 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 18:56:42.589872 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:42.589738    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh" podUID="971fec4f-5a07-4eae-b318-0e9c8f381983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 18:56:43.588461 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:43.588422    4924 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 18:56:43.588883 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:43.588482    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh" podUID="971fec4f-5a07-4eae-b318-0e9c8f381983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 18:56:44.588434 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:44.588394    4924 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 18:56:44.588649 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:44.588453    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh" podUID="971fec4f-5a07-4eae-b318-0e9c8f381983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 18:56:45.588344 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:45.588307    4924 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 18:56:45.588504 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:45.588375    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh" podUID="971fec4f-5a07-4eae-b318-0e9c8f381983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 18:56:45.737798 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:56:45.737759    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:56:46.588684 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:46.588653    4924 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 18:56:46.588854 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:46.588703    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh" podUID="971fec4f-5a07-4eae-b318-0e9c8f381983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 18:56:46.771530 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:46.771491    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:46.982039 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:46.982002    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:56:46.995340 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:46.995313    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:56:47.088526 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:47.088494    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:56:47.588499 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:47.588465    4924 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 18:56:47.588669 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:47.588520    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh" podUID="971fec4f-5a07-4eae-b318-0e9c8f381983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 18:56:48.592577 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:48.592544    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:48.593002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:48.592983    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:48.593633 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:48.593616    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 18:56:51.536465 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:51.536357    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:56.430318 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:56.430280    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:56:59.545851 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:56:59.545812    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:01.337756 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:01.337720    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:06.163404 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:06.163364    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:10.269619 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.269580    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"]
Oct 30 18:57:10.274366 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.274344    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.301048 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.301024    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467dd97e-420c-44ac-a249-17a3deacfb63-istio-token\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.301174 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.301113    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-workload-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.301242 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.301172    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-credential-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.301242 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.301235    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-workload-certs\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.301337 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.301272    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467dd97e-420c-44ac-a249-17a3deacfb63-istio-podinfo\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.301337 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.301292    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-istio-envoy\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.301454 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.301380    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzhtl\" (UniqueName: \"kubernetes.io/projected/467dd97e-420c-44ac-a249-17a3deacfb63-kube-api-access-lzhtl\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.301510 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.301450    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467dd97e-420c-44ac-a249-17a3deacfb63-istiod-ca-cert\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.301567 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.301533    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-istio-data\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.376352 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.376318    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"]
Oct 30 18:57:10.402987 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.402955    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467dd97e-420c-44ac-a249-17a3deacfb63-istiod-ca-cert\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.403192 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.403050    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-istio-data\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.403192 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.403141    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467dd97e-420c-44ac-a249-17a3deacfb63-istio-token\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.403307 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.403251    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-workload-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.403361 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.403315    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-credential-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.403361 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.403352    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-workload-certs\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.403460 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.403401    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467dd97e-420c-44ac-a249-17a3deacfb63-istio-podinfo\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.403460 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.403442    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-istio-envoy\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.403558 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.403474    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzhtl\" (UniqueName: \"kubernetes.io/projected/467dd97e-420c-44ac-a249-17a3deacfb63-kube-api-access-lzhtl\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.403848 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.403821    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467dd97e-420c-44ac-a249-17a3deacfb63-istiod-ca-cert\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.404524 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.404492    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-workload-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.404662 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.404504    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-istio-data\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.404736 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.404710    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-credential-socket\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.404942 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.404912    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-workload-certs\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.407557 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.407511    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467dd97e-420c-44ac-a249-17a3deacfb63-istio-podinfo\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.408635 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.408609    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-istio-envoy\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.459102 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.459054    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467dd97e-420c-44ac-a249-17a3deacfb63-istio-token\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.459491 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.459473    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzhtl\" (UniqueName: \"kubernetes.io/projected/467dd97e-420c-44ac-a249-17a3deacfb63-kube-api-access-lzhtl\") pod \"e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") " pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.589035 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.588958    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:10.890259 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:10.890220    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:11.008824 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:57:11.008735    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467dd97e_420c_44ac_a249_17a3deacfb63.slice/crio-6d442f3484aad27339470a68d59135f628f84240d60b80f9007598cb973a4e85 WatchSource:0}: Error finding container 6d442f3484aad27339470a68d59135f628f84240d60b80f9007598cb973a4e85: Status 404 returned error can't find the container with id 6d442f3484aad27339470a68d59135f628f84240d60b80f9007598cb973a4e85
Oct 30 18:57:11.008962 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:11.008867    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"]
Oct 30 18:57:11.011259 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:11.011227    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:57:11.011358 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:11.011288    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:57:11.011358 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:11.011318    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:57:11.259717 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:11.259630    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz" event={"ID":"467dd97e-420c-44ac-a249-17a3deacfb63","Type":"ContainerStarted","Data":"de74ea692d63df6c2372d51537836a2f9171a9a0890b72e08b0506c6818fd512"}
Oct 30 18:57:11.259717 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:11.259682    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz" event={"ID":"467dd97e-420c-44ac-a249-17a3deacfb63","Type":"ContainerStarted","Data":"6d442f3484aad27339470a68d59135f628f84240d60b80f9007598cb973a4e85"}
Oct 30 18:57:11.372652 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:11.372590    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz" podStartSLOduration=1.3725712749999999 podStartE2EDuration="1.372571275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:11.369383823 +0000 UTC m=+1232.646434268" watchObservedRunningTime="2026-10-30 18:57:11.372571275 +0000 UTC m=+1232.649621721"
Oct 30 18:57:11.590095 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:11.589996    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:12.591227 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:12.591196    4924 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 18:57:12.591596 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:12.591246    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz" podUID="467dd97e-420c-44ac-a249-17a3deacfb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 18:57:13.589605 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:13.589571    4924 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 18:57:13.589809 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:13.589660    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz" podUID="467dd97e-420c-44ac-a249-17a3deacfb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 18:57:14.589816 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:14.589783    4924 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 18:57:14.590273 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:14.589847    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz" podUID="467dd97e-420c-44ac-a249-17a3deacfb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 18:57:15.590348 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:15.590317    4924 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 18:57:15.590758 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:15.590368    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz" podUID="467dd97e-420c-44ac-a249-17a3deacfb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 18:57:15.736017 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:57:15.735970    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:57:15.837338 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:15.837303    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:16.589927 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:16.589899    4924 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 18:57:16.590132 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:16.589946    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz" podUID="467dd97e-420c-44ac-a249-17a3deacfb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 18:57:17.008290 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:17.008250    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"]
Oct 30 18:57:17.008701 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:17.008503    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc" podUID="af172f82-2db0-4251-8669-ae5a1415c606" containerName="istio-proxy" containerID="cri-o://fef22f044647d3caab513979d7fea0d7a59c9bf3255e4f5af66fb59dd702066c" gracePeriod=30
Oct 30 18:57:17.590348 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:17.590315    4924 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 18:57:17.590527 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:17.590376    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz" podUID="467dd97e-420c-44ac-a249-17a3deacfb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 18:57:18.594595 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:18.594558    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:18.595178 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:18.595146    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:18.595958 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:18.595922    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 18:57:20.927019 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:20.926978    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:21.723616 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:21.723581    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"]
Oct 30 18:57:21.729123 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:21.729101    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:57:21.782620 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:21.782572    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"]
Oct 30 18:57:21.823688 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:21.823663    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-3b6ad6\"/\"distinct-w5-3b6ad6-kserve-self-signed-certs\""
Oct 30 18:57:21.852318 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:21.852287    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-dshm\") pod \"distinct-w5-3b6ad6-kserve-94bc474-mbpld\" (UID: \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\") " pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:57:21.852467 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:21.852335    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-home\") pod \"distinct-w5-3b6ad6-kserve-94bc474-mbpld\" (UID: \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\") " pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:57:21.852467 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:21.852365    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-model-cache\") pod \"distinct-w5-3b6ad6-kserve-94bc474-mbpld\" (UID: \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\") " pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:57:21.852467 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:21.852439    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbhg8\" (UniqueName: \"kubernetes.io/projected/f04af09f-b8d2-4b47-b0b3-88eb12adff93-kube-api-access-kbhg8\") pod \"distinct-w5-3b6ad6-kserve-94bc474-mbpld\" (UID: \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\") " pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:57:21.852571 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:21.852489    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-tmp-dir\") pod \"distinct-w5-3b6ad6-kserve-94bc474-mbpld\" (UID: \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\") " pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:57:21.852571 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:21.852539    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f04af09f-b8d2-4b47-b0b3-88eb12adff93-tls-certs\") pod \"distinct-w5-3b6ad6-kserve-94bc474-mbpld\" (UID: \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\") " pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:57:21.883962 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:21.883916    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:21.953411 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:21.953380    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-tmp-dir\") pod \"distinct-w5-3b6ad6-kserve-94bc474-mbpld\" (UID: \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\") " pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:57:21.953880 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:21.953428    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f04af09f-b8d2-4b47-b0b3-88eb12adff93-tls-certs\") pod \"distinct-w5-3b6ad6-kserve-94bc474-mbpld\" (UID: \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\") " pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:57:21.953880 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:21.953537    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-dshm\") pod \"distinct-w5-3b6ad6-kserve-94bc474-mbpld\" (UID: \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\") " pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:57:21.953880 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:21.953580    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-home\") pod \"distinct-w5-3b6ad6-kserve-94bc474-mbpld\" (UID: \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\") " pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:57:21.953880 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:21.953602    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-model-cache\") pod \"distinct-w5-3b6ad6-kserve-94bc474-mbpld\" (UID: \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\") " pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:57:21.953880 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:21.953643    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbhg8\" (UniqueName: \"kubernetes.io/projected/f04af09f-b8d2-4b47-b0b3-88eb12adff93-kube-api-access-kbhg8\") pod \"distinct-w5-3b6ad6-kserve-94bc474-mbpld\" (UID: \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\") " pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:57:21.953880 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:21.953850    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-tmp-dir\") pod \"distinct-w5-3b6ad6-kserve-94bc474-mbpld\" (UID: \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\") " pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:57:21.954129 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:21.953973    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-home\") pod \"distinct-w5-3b6ad6-kserve-94bc474-mbpld\" (UID: \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\") " pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:57:21.954129 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:21.954016    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-model-cache\") pod \"distinct-w5-3b6ad6-kserve-94bc474-mbpld\" (UID: \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\") " pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:57:21.955799 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:21.955777    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-dshm\") pod \"distinct-w5-3b6ad6-kserve-94bc474-mbpld\" (UID: \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\") " pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:57:21.956111 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:21.956091    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f04af09f-b8d2-4b47-b0b3-88eb12adff93-tls-certs\") pod \"distinct-w5-3b6ad6-kserve-94bc474-mbpld\" (UID: \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\") " pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:57:22.012997 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.012924    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbhg8\" (UniqueName: \"kubernetes.io/projected/f04af09f-b8d2-4b47-b0b3-88eb12adff93-kube-api-access-kbhg8\") pod \"distinct-w5-3b6ad6-kserve-94bc474-mbpld\" (UID: \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\") " pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:57:22.042132 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.042110    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:57:22.291367 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.291343    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:57:22.354103 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.354059    4924 generic.go:356] "Generic (PLEG): container finished" podID="af172f82-2db0-4251-8669-ae5a1415c606" containerID="fef22f044647d3caab513979d7fea0d7a59c9bf3255e4f5af66fb59dd702066c" exitCode=0
Oct 30 18:57:22.354290 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.354117    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc" event={"ID":"af172f82-2db0-4251-8669-ae5a1415c606","Type":"ContainerDied","Data":"fef22f044647d3caab513979d7fea0d7a59c9bf3255e4f5af66fb59dd702066c"}
Oct 30 18:57:22.354290 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.354174    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc" event={"ID":"af172f82-2db0-4251-8669-ae5a1415c606","Type":"ContainerDied","Data":"46529fefce8a54e0888c535d3db0f56e947848796709d71776ff7abafd246119"}
Oct 30 18:57:22.354290 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.354196    4924 scope.go:122] "RemoveContainer" containerID="fef22f044647d3caab513979d7fea0d7a59c9bf3255e4f5af66fb59dd702066c"
Oct 30 18:57:22.354290 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.354127    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"
Oct 30 18:57:22.359520 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.359500    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af172f82-2db0-4251-8669-ae5a1415c606-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af172f82-2db0-4251-8669-ae5a1415c606-istiod-ca-cert\") pod \"af172f82-2db0-4251-8669-ae5a1415c606\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") "
Oct 30 18:57:22.359640 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.359542    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-credential-socket\") pod \"af172f82-2db0-4251-8669-ae5a1415c606\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") "
Oct 30 18:57:22.359640 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.359574    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af172f82-2db0-4251-8669-ae5a1415c606-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af172f82-2db0-4251-8669-ae5a1415c606-istio-podinfo\") pod \"af172f82-2db0-4251-8669-ae5a1415c606\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") "
Oct 30 18:57:22.359640 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.359621    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-workload-socket\") pod \"af172f82-2db0-4251-8669-ae5a1415c606\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") "
Oct 30 18:57:22.359807 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.359735    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af172f82-2db0-4251-8669-ae5a1415c606-istio-token\" (UniqueName: \"kubernetes.io/projected/af172f82-2db0-4251-8669-ae5a1415c606-istio-token\") pod \"af172f82-2db0-4251-8669-ae5a1415c606\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") "
Oct 30 18:57:22.359807 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.359777    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-istio-envoy\") pod \"af172f82-2db0-4251-8669-ae5a1415c606\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") "
Oct 30 18:57:22.359909 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.359804    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-credential-socket" pod "af172f82-2db0-4251-8669-ae5a1415c606" (UID: "af172f82-2db0-4251-8669-ae5a1415c606"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:22.359975 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.359958    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-istio-data\") pod \"af172f82-2db0-4251-8669-ae5a1415c606\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") "
Oct 30 18:57:22.360026 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.360001    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af172f82-2db0-4251-8669-ae5a1415c606-kube-api-access-6xfg7\" (UniqueName: \"kubernetes.io/projected/af172f82-2db0-4251-8669-ae5a1415c606-kube-api-access-6xfg7\") pod \"af172f82-2db0-4251-8669-ae5a1415c606\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") "
Oct 30 18:57:22.360026 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.360006    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-workload-socket" pod "af172f82-2db0-4251-8669-ae5a1415c606" (UID: "af172f82-2db0-4251-8669-ae5a1415c606"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:22.360231 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.360023    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af172f82-2db0-4251-8669-ae5a1415c606-istiod-ca-cert" pod "af172f82-2db0-4251-8669-ae5a1415c606" (UID: "af172f82-2db0-4251-8669-ae5a1415c606"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:22.360231 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.360028    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-workload-certs\") pod \"af172f82-2db0-4251-8669-ae5a1415c606\" (UID: \"af172f82-2db0-4251-8669-ae5a1415c606\") "
Oct 30 18:57:22.360410 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.360258    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-istio-data" pod "af172f82-2db0-4251-8669-ae5a1415c606" (UID: "af172f82-2db0-4251-8669-ae5a1415c606"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:22.360471 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.360405    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-workload-certs" pod "af172f82-2db0-4251-8669-ae5a1415c606" (UID: "af172f82-2db0-4251-8669-ae5a1415c606"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:22.360657 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.360623    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af172f82-2db0-4251-8669-ae5a1415c606-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:22.360657 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.360651    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:22.360761 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.360669    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:22.360761 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.360684    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:22.360761 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.360700    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:22.362202 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.362179    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af172f82-2db0-4251-8669-ae5a1415c606-kube-api-access-6xfg7" pod "af172f82-2db0-4251-8669-ae5a1415c606" (UID: "af172f82-2db0-4251-8669-ae5a1415c606"). InnerVolumeSpecName "kube-api-access-6xfg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:22.362420 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.362399    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af172f82-2db0-4251-8669-ae5a1415c606-istio-token" pod "af172f82-2db0-4251-8669-ae5a1415c606" (UID: "af172f82-2db0-4251-8669-ae5a1415c606"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:22.362420 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.362408    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-istio-envoy" pod "af172f82-2db0-4251-8669-ae5a1415c606" (UID: "af172f82-2db0-4251-8669-ae5a1415c606"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:22.362693 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.362672    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af172f82-2db0-4251-8669-ae5a1415c606-istio-podinfo" pod "af172f82-2db0-4251-8669-ae5a1415c606" (UID: "af172f82-2db0-4251-8669-ae5a1415c606"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 18:57:22.364592 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.364573    4924 scope.go:122] "RemoveContainer" containerID="fef22f044647d3caab513979d7fea0d7a59c9bf3255e4f5af66fb59dd702066c"
Oct 30 18:57:22.364882 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:57:22.364860    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fef22f044647d3caab513979d7fea0d7a59c9bf3255e4f5af66fb59dd702066c\": container with ID starting with fef22f044647d3caab513979d7fea0d7a59c9bf3255e4f5af66fb59dd702066c not found: ID does not exist" containerID="fef22f044647d3caab513979d7fea0d7a59c9bf3255e4f5af66fb59dd702066c"
Oct 30 18:57:22.364928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.364892    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fef22f044647d3caab513979d7fea0d7a59c9bf3255e4f5af66fb59dd702066c"} err="failed to get container status \"fef22f044647d3caab513979d7fea0d7a59c9bf3255e4f5af66fb59dd702066c\": rpc error: code = NotFound desc = could not find container \"fef22f044647d3caab513979d7fea0d7a59c9bf3255e4f5af66fb59dd702066c\": container with ID starting with fef22f044647d3caab513979d7fea0d7a59c9bf3255e4f5af66fb59dd702066c not found: ID does not exist"
Oct 30 18:57:22.451881 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.451856    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"]
Oct 30 18:57:22.452964 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:57:22.452940    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf04af09f_b8d2_4b47_b0b3_88eb12adff93.slice/crio-08a62a56a121b9500a464e8f396f20f26e97af55037d97599bdbf298ffe22312 WatchSource:0}: Error finding container 08a62a56a121b9500a464e8f396f20f26e97af55037d97599bdbf298ffe22312: Status 404 returned error can't find the container with id 08a62a56a121b9500a464e8f396f20f26e97af55037d97599bdbf298ffe22312
Oct 30 18:57:22.462314 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.462288    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af172f82-2db0-4251-8669-ae5a1415c606-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:22.462401 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.462327    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af172f82-2db0-4251-8669-ae5a1415c606-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:22.462401 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.462342    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af172f82-2db0-4251-8669-ae5a1415c606-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:22.462401 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.462357    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xfg7\" (UniqueName: \"kubernetes.io/projected/af172f82-2db0-4251-8669-ae5a1415c606-kube-api-access-6xfg7\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:22.770928 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.770890    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"]
Oct 30 18:57:22.777561 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.777526    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9pgfc"]
Oct 30 18:57:22.834321 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:22.834287    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af172f82-2db0-4251-8669-ae5a1415c606" path="/var/lib/kubelet/pods/af172f82-2db0-4251-8669-ae5a1415c606/volumes"
Oct 30 18:57:23.364622 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:23.364591    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld" event={"ID":"f04af09f-b8d2-4b47-b0b3-88eb12adff93","Type":"ContainerStarted","Data":"8df63936a23a9397c6a763772d2f3e8852effa7844bbaef26e08152f8dc3523b"}
Oct 30 18:57:23.365013 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:23.364635    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld" event={"ID":"f04af09f-b8d2-4b47-b0b3-88eb12adff93","Type":"ContainerStarted","Data":"08a62a56a121b9500a464e8f396f20f26e97af55037d97599bdbf298ffe22312"}
Oct 30 18:57:23.474107 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:23.474043    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld" podStartSLOduration=2.474031331 podStartE2EDuration="2.474031331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:23.47193871 +0000 UTC m=+1244.748989166" watchObservedRunningTime="2026-10-30 18:57:23.474031331 +0000 UTC m=+1244.751081777"
Oct 30 18:57:24.042825 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:24.042792    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:57:25.662375 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:25.662340    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:30.551992 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:30.551946    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:32.043157 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:32.043042    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:57:32.056466 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:32.056436    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:57:32.437551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:32.437520    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:57:33.118964 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.118928    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"]
Oct 30 18:57:33.119861 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.119837    4924 cpu_manager.go:414] "containerMap: removing container" podUID="af172f82-2db0-4251-8669-ae5a1415c606" containerName="istio-proxy"
Oct 30 18:57:33.119929 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.119866    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af172f82-2db0-4251-8669-ae5a1415c606" containerName="istio-proxy"
Oct 30 18:57:33.120032 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.120018    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="af172f82-2db0-4251-8669-ae5a1415c606" containerName="istio-proxy"
Oct 30 18:57:33.124276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.124251    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:57:33.175745 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.175694    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"]
Oct 30 18:57:33.180654 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.180620    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a1617e\"/\"distinct-2-w3-a1617e-kserve-self-signed-certs\""
Oct 30 18:57:33.193431 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.193401    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-home\") pod \"distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt\" (UID: \"955e98b9-864b-4636-9511-8b69f981eaa7\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:57:33.193558 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.193465    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-model-cache\") pod \"distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt\" (UID: \"955e98b9-864b-4636-9511-8b69f981eaa7\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:57:33.193665 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.193646    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-dshm\") pod \"distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt\" (UID: \"955e98b9-864b-4636-9511-8b69f981eaa7\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:57:33.193788 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.193771    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955e98b9-864b-4636-9511-8b69f981eaa7-tls-certs\") pod \"distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt\" (UID: \"955e98b9-864b-4636-9511-8b69f981eaa7\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:57:33.193857 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.193816    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dldgw\" (UniqueName: \"kubernetes.io/projected/955e98b9-864b-4636-9511-8b69f981eaa7-kube-api-access-dldgw\") pod \"distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt\" (UID: \"955e98b9-864b-4636-9511-8b69f981eaa7\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:57:33.193917 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.193862    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-tmp-dir\") pod \"distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt\" (UID: \"955e98b9-864b-4636-9511-8b69f981eaa7\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:57:33.269687 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.269650    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:33.295142 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.295105    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955e98b9-864b-4636-9511-8b69f981eaa7-tls-certs\") pod \"distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt\" (UID: \"955e98b9-864b-4636-9511-8b69f981eaa7\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:57:33.295335 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.295164    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dldgw\" (UniqueName: \"kubernetes.io/projected/955e98b9-864b-4636-9511-8b69f981eaa7-kube-api-access-dldgw\") pod \"distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt\" (UID: \"955e98b9-864b-4636-9511-8b69f981eaa7\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:57:33.295335 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.295216    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-tmp-dir\") pod \"distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt\" (UID: \"955e98b9-864b-4636-9511-8b69f981eaa7\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:57:33.295469 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.295351    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-home\") pod \"distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt\" (UID: \"955e98b9-864b-4636-9511-8b69f981eaa7\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:57:33.295469 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.295387    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-model-cache\") pod \"distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt\" (UID: \"955e98b9-864b-4636-9511-8b69f981eaa7\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:57:33.295469 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.295464    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-dshm\") pod \"distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt\" (UID: \"955e98b9-864b-4636-9511-8b69f981eaa7\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:57:33.296353 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.296327    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-home\") pod \"distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt\" (UID: \"955e98b9-864b-4636-9511-8b69f981eaa7\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:57:33.296541 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.296364    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-model-cache\") pod \"distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt\" (UID: \"955e98b9-864b-4636-9511-8b69f981eaa7\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:57:33.296669 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.296419    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-tmp-dir\") pod \"distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt\" (UID: \"955e98b9-864b-4636-9511-8b69f981eaa7\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:57:33.298491 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.298470    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-dshm\") pod \"distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt\" (UID: \"955e98b9-864b-4636-9511-8b69f981eaa7\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:57:33.299959 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.299943    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955e98b9-864b-4636-9511-8b69f981eaa7-tls-certs\") pod \"distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt\" (UID: \"955e98b9-864b-4636-9511-8b69f981eaa7\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:57:33.351974 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.351947    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dldgw\" (UniqueName: \"kubernetes.io/projected/955e98b9-864b-4636-9511-8b69f981eaa7-kube-api-access-dldgw\") pod \"distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt\" (UID: \"955e98b9-864b-4636-9511-8b69f981eaa7\") " pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:57:33.440281 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.440199    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:57:33.810530 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:57:33.810488    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod955e98b9_864b_4636_9511_8b69f981eaa7.slice/crio-a80db07cbf793cdcc54b890292c9297d9f7964d55655a5dbde6515ede05583aa WatchSource:0}: Error finding container a80db07cbf793cdcc54b890292c9297d9f7964d55655a5dbde6515ede05583aa: Status 404 returned error can't find the container with id a80db07cbf793cdcc54b890292c9297d9f7964d55655a5dbde6515ede05583aa
Oct 30 18:57:33.810907 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:33.810878    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"]
Oct 30 18:57:34.439606 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:34.439564    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt" event={"ID":"955e98b9-864b-4636-9511-8b69f981eaa7","Type":"ContainerStarted","Data":"38c70d190e6fc191ab9f19230eb117760419769282ceed555089a66a0610698d"}
Oct 30 18:57:34.439995 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:34.439614    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt" event={"ID":"955e98b9-864b-4636-9511-8b69f981eaa7","Type":"ContainerStarted","Data":"a80db07cbf793cdcc54b890292c9297d9f7964d55655a5dbde6515ede05583aa"}
Oct 30 18:57:34.440755 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:34.440741    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:57:34.550192 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:34.550140    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt" podStartSLOduration=1.550127059 podStartE2EDuration="1.550127059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:34.54944007 +0000 UTC m=+1255.826490516" watchObservedRunningTime="2026-10-30 18:57:34.550127059 +0000 UTC m=+1255.827177505"
Oct 30 18:57:35.251029 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:35.250994    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:35.444448 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:35.444413    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:38.446725 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:38.446692    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de3eb91-xtpb2"]
Oct 30 18:57:38.453386 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:38.453361    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de3eb91-xtpb2"
Oct 30 18:57:38.505415 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:38.505387    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de3eb91-xtpb2"]
Oct 30 18:57:38.547207 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:38.547178    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:57:38.547396 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:38.547378    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:57:38.547441 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:38.547398    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 18:57:38.550553 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:38.550538    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:57:38.563878 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:38.563857    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67125a81-e9f2-48c6-b024-0d98c501f112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8de3eb91-xtpb2\" (UID: \"67125a81-e9f2-48c6-b024-0d98c501f112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de3eb91-xtpb2"
Oct 30 18:57:38.564009 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:38.563988    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67125a81-e9f2-48c6-b024-0d98c501f112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8de3eb91-xtpb2\" (UID: \"67125a81-e9f2-48c6-b024-0d98c501f112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de3eb91-xtpb2"
Oct 30 18:57:38.564173 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:38.564155    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg96x\" (UniqueName: \"kubernetes.io/projected/67125a81-e9f2-48c6-b024-0d98c501f112-kube-api-access-tg96x\") pod \"maas-api-revoke-sub-8de3eb91-xtpb2\" (UID: \"67125a81-e9f2-48c6-b024-0d98c501f112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de3eb91-xtpb2"
Oct 30 18:57:38.664798 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:38.664764    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg96x\" (UniqueName: \"kubernetes.io/projected/67125a81-e9f2-48c6-b024-0d98c501f112-kube-api-access-tg96x\") pod \"maas-api-revoke-sub-8de3eb91-xtpb2\" (UID: \"67125a81-e9f2-48c6-b024-0d98c501f112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de3eb91-xtpb2"
Oct 30 18:57:38.664946 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:38.664857    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67125a81-e9f2-48c6-b024-0d98c501f112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8de3eb91-xtpb2\" (UID: \"67125a81-e9f2-48c6-b024-0d98c501f112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de3eb91-xtpb2"
Oct 30 18:57:38.665012 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:38.664981    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67125a81-e9f2-48c6-b024-0d98c501f112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8de3eb91-xtpb2\" (UID: \"67125a81-e9f2-48c6-b024-0d98c501f112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de3eb91-xtpb2"
Oct 30 18:57:38.665713 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:38.665691    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67125a81-e9f2-48c6-b024-0d98c501f112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8de3eb91-xtpb2\" (UID: \"67125a81-e9f2-48c6-b024-0d98c501f112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de3eb91-xtpb2"
Oct 30 18:57:38.667217 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:38.667196    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67125a81-e9f2-48c6-b024-0d98c501f112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8de3eb91-xtpb2\" (UID: \"67125a81-e9f2-48c6-b024-0d98c501f112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de3eb91-xtpb2"
Oct 30 18:57:38.719969 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:38.719917    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg96x\" (UniqueName: \"kubernetes.io/projected/67125a81-e9f2-48c6-b024-0d98c501f112-kube-api-access-tg96x\") pod \"maas-api-revoke-sub-8de3eb91-xtpb2\" (UID: \"67125a81-e9f2-48c6-b024-0d98c501f112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de3eb91-xtpb2"
Oct 30 18:57:38.769180 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:38.769159    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de3eb91-xtpb2"
Oct 30 18:57:39.136184 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:57:39.136152    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67125a81_e9f2_48c6_b024_0d98c501f112.slice/crio-55a0e0c122aa8158ddcaf8c8dbacb06741d0e4bb088cc316747446ea40f04b79 WatchSource:0}: Error finding container 55a0e0c122aa8158ddcaf8c8dbacb06741d0e4bb088cc316747446ea40f04b79: Status 404 returned error can't find the container with id 55a0e0c122aa8158ddcaf8c8dbacb06741d0e4bb088cc316747446ea40f04b79
Oct 30 18:57:39.137659 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:39.137630    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de3eb91-xtpb2"]
Oct 30 18:57:39.477534 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:39.477502    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de3eb91-xtpb2" event={"ID":"67125a81-e9f2-48c6-b024-0d98c501f112","Type":"ContainerStarted","Data":"2d5cbf2ac72e53278cf3e8912f093e40e806619e00e49c2b486a593b8d949485"}
Oct 30 18:57:39.477868 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:39.477552    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de3eb91-xtpb2" event={"ID":"67125a81-e9f2-48c6-b024-0d98c501f112","Type":"ContainerStarted","Data":"55a0e0c122aa8158ddcaf8c8dbacb06741d0e4bb088cc316747446ea40f04b79"}
Oct 30 18:57:39.589261 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:39.589175    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de3eb91-xtpb2" podStartSLOduration=1.589159993 podStartE2EDuration="1.589159993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:39.584673045 +0000 UTC m=+1260.861723473" watchObservedRunningTime="2026-10-30 18:57:39.589159993 +0000 UTC m=+1260.866210435"
Oct 30 18:57:40.269059 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.268975    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"]
Oct 30 18:57:40.273253 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.273233    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.320690 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.320646    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:40.376870 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.376838    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"]
Oct 30 18:57:40.384785 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.384737    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-envoy\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.384927 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.384785    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-token\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.384978 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.384944    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-credential-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.385029 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.385008    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-workload-certs\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.385104 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.385054    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-data\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.385164 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.385147    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-podinfo\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.385202 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.385185    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-workload-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.385253 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.385237    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzjth\" (UniqueName: \"kubernetes.io/projected/34dfd0fe-b500-4df5-b191-6085a5d8dde5-kube-api-access-lzjth\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.385289 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.385279    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istiod-ca-cert\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.486131 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.486100    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-envoy\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.486595 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.486138    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-token\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.486595 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.486214    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-credential-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.486595 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.486263    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-workload-certs\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.486595 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.486295    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-data\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.486595 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.486354    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-podinfo\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.486595 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.486390    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-workload-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.486595 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.486440    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzjth\" (UniqueName: \"kubernetes.io/projected/34dfd0fe-b500-4df5-b191-6085a5d8dde5-kube-api-access-lzjth\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.486595 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.486480    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istiod-ca-cert\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.487029 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.486721    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-credential-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.487029 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.486746    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-data\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.487029 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.486969    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-workload-socket\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.487253 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.487235    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-workload-certs\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.487365 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.487341    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istiod-ca-cert\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.487433 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.487415    4924 generic.go:356] "Generic (PLEG): container finished" podID="67125a81-e9f2-48c6-b024-0d98c501f112" containerID="2d5cbf2ac72e53278cf3e8912f093e40e806619e00e49c2b486a593b8d949485" exitCode=0
Oct 30 18:57:40.487560 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.487534    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de3eb91-xtpb2" event={"ID":"67125a81-e9f2-48c6-b024-0d98c501f112","Type":"ContainerDied","Data":"2d5cbf2ac72e53278cf3e8912f093e40e806619e00e49c2b486a593b8d949485"}
Oct 30 18:57:40.488603 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.488579    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-envoy\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.488923 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.488902    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-podinfo\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.540719 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.540659    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-token\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.541026 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.541008    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzjth\" (UniqueName: \"kubernetes.io/projected/34dfd0fe-b500-4df5-b191-6085a5d8dde5-kube-api-access-lzjth\") pod \"e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") " pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:40.587406 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:40.587373    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:41.019334 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:41.019304    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"]
Oct 30 18:57:41.020613 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:57:41.020588    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34dfd0fe_b500_4df5_b191_6085a5d8dde5.slice/crio-cd580ed539c16295dbf7466b16eeaf0eeea057f15bc2cf73f95867a064450445 WatchSource:0}: Error finding container cd580ed539c16295dbf7466b16eeaf0eeea057f15bc2cf73f95867a064450445: Status 404 returned error can't find the container with id cd580ed539c16295dbf7466b16eeaf0eeea057f15bc2cf73f95867a064450445
Oct 30 18:57:41.023543 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:41.023510    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:57:41.023620 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:41.023577    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:57:41.023620 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:41.023605    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:57:41.496278 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:41.496234    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2" event={"ID":"34dfd0fe-b500-4df5-b191-6085a5d8dde5","Type":"ContainerStarted","Data":"6a7aa424319bce6c1197a1f465f648982b85daafa1d99bfbb22483f3a46178df"}
Oct 30 18:57:41.496813 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:41.496297    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2" event={"ID":"34dfd0fe-b500-4df5-b191-6085a5d8dde5","Type":"ContainerStarted","Data":"cd580ed539c16295dbf7466b16eeaf0eeea057f15bc2cf73f95867a064450445"}
Oct 30 18:57:41.588139 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:41.588097    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:41.671231 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:41.671182    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2" podStartSLOduration=1.6711658489999999 podStartE2EDuration="1.671165849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:41.668718859 +0000 UTC m=+1262.945769301" watchObservedRunningTime="2026-10-30 18:57:41.671165849 +0000 UTC m=+1262.948216293"
Oct 30 18:57:41.674182 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:41.674163    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de3eb91-xtpb2"
Oct 30 18:57:41.698763 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:41.698731    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67125a81-e9f2-48c6-b024-0d98c501f112-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67125a81-e9f2-48c6-b024-0d98c501f112-maas-api-openshift-ca\") pod \"67125a81-e9f2-48c6-b024-0d98c501f112\" (UID: \"67125a81-e9f2-48c6-b024-0d98c501f112\") "
Oct 30 18:57:41.698906 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:41.698784    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67125a81-e9f2-48c6-b024-0d98c501f112-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67125a81-e9f2-48c6-b024-0d98c501f112-maas-api-xks-ca\") pod \"67125a81-e9f2-48c6-b024-0d98c501f112\" (UID: \"67125a81-e9f2-48c6-b024-0d98c501f112\") "
Oct 30 18:57:41.698979 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:41.698948    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67125a81-e9f2-48c6-b024-0d98c501f112-kube-api-access-tg96x\" (UniqueName: \"kubernetes.io/projected/67125a81-e9f2-48c6-b024-0d98c501f112-kube-api-access-tg96x\") pod \"67125a81-e9f2-48c6-b024-0d98c501f112\" (UID: \"67125a81-e9f2-48c6-b024-0d98c501f112\") "
Oct 30 18:57:41.699087 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:41.699041    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67125a81-e9f2-48c6-b024-0d98c501f112-maas-api-openshift-ca" pod "67125a81-e9f2-48c6-b024-0d98c501f112" (UID: "67125a81-e9f2-48c6-b024-0d98c501f112"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:41.699323 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:41.699305    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67125a81-e9f2-48c6-b024-0d98c501f112-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:41.700859 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:41.700834    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67125a81-e9f2-48c6-b024-0d98c501f112-maas-api-xks-ca" pod "67125a81-e9f2-48c6-b024-0d98c501f112" (UID: "67125a81-e9f2-48c6-b024-0d98c501f112"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:57:41.701098 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:41.701076    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67125a81-e9f2-48c6-b024-0d98c501f112-kube-api-access-tg96x" pod "67125a81-e9f2-48c6-b024-0d98c501f112" (UID: "67125a81-e9f2-48c6-b024-0d98c501f112"). InnerVolumeSpecName "kube-api-access-tg96x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:41.800910 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:41.800835    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67125a81-e9f2-48c6-b024-0d98c501f112-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:41.800910 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:41.800874    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg96x\" (UniqueName: \"kubernetes.io/projected/67125a81-e9f2-48c6-b024-0d98c501f112-kube-api-access-tg96x\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:42.529264 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:42.529236    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de3eb91-xtpb2"
Oct 30 18:57:42.529695 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:42.529230    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de3eb91-xtpb2" event={"ID":"67125a81-e9f2-48c6-b024-0d98c501f112","Type":"ContainerDied","Data":"55a0e0c122aa8158ddcaf8c8dbacb06741d0e4bb088cc316747446ea40f04b79"}
Oct 30 18:57:42.529695 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:42.529335    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55a0e0c122aa8158ddcaf8c8dbacb06741d0e4bb088cc316747446ea40f04b79"
Oct 30 18:57:42.589259 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:42.589231    4924 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 18:57:42.589402 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:42.589279    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2" podUID="34dfd0fe-b500-4df5-b191-6085a5d8dde5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 18:57:43.440546 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:43.440512    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:57:43.453414 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:43.453386    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:57:43.548397 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:43.548369    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:57:43.588148 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:43.588121    4924 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 18:57:43.588282 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:43.588165    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2" podUID="34dfd0fe-b500-4df5-b191-6085a5d8dde5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 18:57:44.588270 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:44.588237    4924 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 18:57:44.588696 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:44.588299    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2" podUID="34dfd0fe-b500-4df5-b191-6085a5d8dde5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 18:57:45.243866 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:45.243826    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:45.588769 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:45.588688    4924 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 18:57:45.588769 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:45.588748    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2" podUID="34dfd0fe-b500-4df5-b191-6085a5d8dde5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 18:57:45.732874 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:57:45.732836    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:57:46.587778 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:46.587748    4924 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 18:57:46.587948 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:46.587803    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2" podUID="34dfd0fe-b500-4df5-b191-6085a5d8dde5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 18:57:47.592103 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:47.590642    4924 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 18:57:47.592103 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:47.590735    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2" podUID="34dfd0fe-b500-4df5-b191-6085a5d8dde5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 18:57:48.625084 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:48.619898    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:48.625084 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:48.621297    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:48.642086 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:48.633281    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 18:57:50.765708 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:50.765673    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:52.792651 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:52.792607    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b44d0e1-86vs5"]
Oct 30 18:57:52.793346 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:52.793324    4924 cpu_manager.go:414] "containerMap: removing container" podUID="67125a81-e9f2-48c6-b024-0d98c501f112" containerName="revoke-keys"
Oct 30 18:57:52.793346 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:52.793347    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67125a81-e9f2-48c6-b024-0d98c501f112" containerName="revoke-keys"
Oct 30 18:57:52.793537 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:52.793525    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="67125a81-e9f2-48c6-b024-0d98c501f112" containerName="revoke-keys"
Oct 30 18:57:52.798168 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:52.798153    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b44d0e1-86vs5"
Oct 30 18:57:52.850612 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:52.850587    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b44d0e1-86vs5"]
Oct 30 18:57:52.872219 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:52.872188    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca973523-e0ce-42d4-9d1e-bd0eab35c776-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b44d0e1-86vs5\" (UID: \"ca973523-e0ce-42d4-9d1e-bd0eab35c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b44d0e1-86vs5"
Oct 30 18:57:52.872364 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:52.872344    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7qp9\" (UniqueName: \"kubernetes.io/projected/ca973523-e0ce-42d4-9d1e-bd0eab35c776-kube-api-access-f7qp9\") pod \"maas-api-revoke-sub-6b44d0e1-86vs5\" (UID: \"ca973523-e0ce-42d4-9d1e-bd0eab35c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b44d0e1-86vs5"
Oct 30 18:57:52.872426 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:52.872380    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca973523-e0ce-42d4-9d1e-bd0eab35c776-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b44d0e1-86vs5\" (UID: \"ca973523-e0ce-42d4-9d1e-bd0eab35c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b44d0e1-86vs5"
Oct 30 18:57:52.891742 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:52.891714    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:57:52.891742 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:52.891728    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:57:52.891902 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:52.891750    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:57:52.894586 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:52.894573    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 18:57:52.973848 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:52.973821    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca973523-e0ce-42d4-9d1e-bd0eab35c776-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b44d0e1-86vs5\" (UID: \"ca973523-e0ce-42d4-9d1e-bd0eab35c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b44d0e1-86vs5"
Oct 30 18:57:52.973975 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:52.973902    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7qp9\" (UniqueName: \"kubernetes.io/projected/ca973523-e0ce-42d4-9d1e-bd0eab35c776-kube-api-access-f7qp9\") pod \"maas-api-revoke-sub-6b44d0e1-86vs5\" (UID: \"ca973523-e0ce-42d4-9d1e-bd0eab35c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b44d0e1-86vs5"
Oct 30 18:57:52.973975 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:52.973937    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca973523-e0ce-42d4-9d1e-bd0eab35c776-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b44d0e1-86vs5\" (UID: \"ca973523-e0ce-42d4-9d1e-bd0eab35c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b44d0e1-86vs5"
Oct 30 18:57:52.974670 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:52.974647    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca973523-e0ce-42d4-9d1e-bd0eab35c776-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b44d0e1-86vs5\" (UID: \"ca973523-e0ce-42d4-9d1e-bd0eab35c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b44d0e1-86vs5"
Oct 30 18:57:52.976172 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:52.976153    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca973523-e0ce-42d4-9d1e-bd0eab35c776-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b44d0e1-86vs5\" (UID: \"ca973523-e0ce-42d4-9d1e-bd0eab35c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b44d0e1-86vs5"
Oct 30 18:57:53.028343 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:53.028318    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7qp9\" (UniqueName: \"kubernetes.io/projected/ca973523-e0ce-42d4-9d1e-bd0eab35c776-kube-api-access-f7qp9\") pod \"maas-api-revoke-sub-6b44d0e1-86vs5\" (UID: \"ca973523-e0ce-42d4-9d1e-bd0eab35c776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b44d0e1-86vs5"
Oct 30 18:57:53.111638 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:53.111614    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b44d0e1-86vs5"
Oct 30 18:57:53.484831 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:57:53.484801    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca973523_e0ce_42d4_9d1e_bd0eab35c776.slice/crio-e5113bea08aff8a9dce197bcfdaf7bcbeeba41eb78ab7d7c7786d0e5d56aa3ed WatchSource:0}: Error finding container e5113bea08aff8a9dce197bcfdaf7bcbeeba41eb78ab7d7c7786d0e5d56aa3ed: Status 404 returned error can't find the container with id e5113bea08aff8a9dce197bcfdaf7bcbeeba41eb78ab7d7c7786d0e5d56aa3ed
Oct 30 18:57:53.485750 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:53.485726    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b44d0e1-86vs5"]
Oct 30 18:57:53.665417 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:53.665381    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b44d0e1-86vs5" event={"ID":"ca973523-e0ce-42d4-9d1e-bd0eab35c776","Type":"ContainerStarted","Data":"e5113bea08aff8a9dce197bcfdaf7bcbeeba41eb78ab7d7c7786d0e5d56aa3ed"}
Oct 30 18:57:54.674129 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:54.674033    4924 generic.go:356] "Generic (PLEG): container finished" podID="ca973523-e0ce-42d4-9d1e-bd0eab35c776" containerID="19314710ef7203226e765b9f40974387877c9d2e6198c6eb1d3961b4d750384a" exitCode=0
Oct 30 18:57:54.674473 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:54.674219    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b44d0e1-86vs5" event={"ID":"ca973523-e0ce-42d4-9d1e-bd0eab35c776","Type":"ContainerDied","Data":"19314710ef7203226e765b9f40974387877c9d2e6198c6eb1d3961b4d750384a"}
Oct 30 18:57:55.264500 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.264465    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"]
Oct 30 18:57:55.269131 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.269109    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.369577 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.369538    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"]
Oct 30 18:57:55.415443 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.415410    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af3a685-9810-41d3-9ba3-97e510ce2d96-istiod-ca-cert\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.415606 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.415552    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-workload-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.415606 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.415600    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-workload-certs\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.415689 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.415618    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sh66\" (UniqueName: \"kubernetes.io/projected/7af3a685-9810-41d3-9ba3-97e510ce2d96-kube-api-access-8sh66\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.415689 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.415644    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-podinfo\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.415689 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.415675    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-token\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.415783 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.415750    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-data\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.415818 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.415793    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-envoy\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.415911 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.415889    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-credential-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.498078 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.498045    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:57:55.517244 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.517156    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-credential-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.517428 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.517249    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af3a685-9810-41d3-9ba3-97e510ce2d96-istiod-ca-cert\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.517428 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.517366    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-workload-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.517566 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.517432    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-workload-certs\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.517566 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.517485    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sh66\" (UniqueName: \"kubernetes.io/projected/7af3a685-9810-41d3-9ba3-97e510ce2d96-kube-api-access-8sh66\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.517566 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.517524    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-podinfo\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.517721 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.517652    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-credential-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.517941 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.517914    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-workload-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.518360 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.517956    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-workload-certs\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.518360 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.518139    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-token\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.518360 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.518187    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af3a685-9810-41d3-9ba3-97e510ce2d96-istiod-ca-cert\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.518360 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.518245    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-data\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.518360 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.518321    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-envoy\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.519405 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.519362    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-data\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.520950 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.520920    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-podinfo\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.521502 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.521478    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-envoy\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.572756 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.572721    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sh66\" (UniqueName: \"kubernetes.io/projected/7af3a685-9810-41d3-9ba3-97e510ce2d96-kube-api-access-8sh66\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.573397 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.573373    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-token\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.582439 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.582417    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:55.823517 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.823490    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b44d0e1-86vs5"
Oct 30 18:57:55.923660 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.923637    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca973523-e0ce-42d4-9d1e-bd0eab35c776-kube-api-access-f7qp9\" (UniqueName: \"kubernetes.io/projected/ca973523-e0ce-42d4-9d1e-bd0eab35c776-kube-api-access-f7qp9\") pod \"ca973523-e0ce-42d4-9d1e-bd0eab35c776\" (UID: \"ca973523-e0ce-42d4-9d1e-bd0eab35c776\") "
Oct 30 18:57:55.923806 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.923739    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca973523-e0ce-42d4-9d1e-bd0eab35c776-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca973523-e0ce-42d4-9d1e-bd0eab35c776-maas-api-xks-ca\") pod \"ca973523-e0ce-42d4-9d1e-bd0eab35c776\" (UID: \"ca973523-e0ce-42d4-9d1e-bd0eab35c776\") "
Oct 30 18:57:55.923806 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.923795    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca973523-e0ce-42d4-9d1e-bd0eab35c776-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca973523-e0ce-42d4-9d1e-bd0eab35c776-maas-api-openshift-ca\") pod \"ca973523-e0ce-42d4-9d1e-bd0eab35c776\" (UID: \"ca973523-e0ce-42d4-9d1e-bd0eab35c776\") "
Oct 30 18:57:55.924220 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.924195    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca973523-e0ce-42d4-9d1e-bd0eab35c776-maas-api-openshift-ca" pod "ca973523-e0ce-42d4-9d1e-bd0eab35c776" (UID: "ca973523-e0ce-42d4-9d1e-bd0eab35c776"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:55.924481 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.924459    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca973523-e0ce-42d4-9d1e-bd0eab35c776-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:55.925787 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.925761    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca973523-e0ce-42d4-9d1e-bd0eab35c776-kube-api-access-f7qp9" pod "ca973523-e0ce-42d4-9d1e-bd0eab35c776" (UID: "ca973523-e0ce-42d4-9d1e-bd0eab35c776"). InnerVolumeSpecName "kube-api-access-f7qp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:55.925882 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:55.925777    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca973523-e0ce-42d4-9d1e-bd0eab35c776-maas-api-xks-ca" pod "ca973523-e0ce-42d4-9d1e-bd0eab35c776" (UID: "ca973523-e0ce-42d4-9d1e-bd0eab35c776"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:57:56.025607 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:56.025573    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7qp9\" (UniqueName: \"kubernetes.io/projected/ca973523-e0ce-42d4-9d1e-bd0eab35c776-kube-api-access-f7qp9\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:56.025607 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:56.025601    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca973523-e0ce-42d4-9d1e-bd0eab35c776-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:56.050074 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:57:56.050039    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af3a685_9810_41d3_9ba3_97e510ce2d96.slice/crio-606693a8796bc424dd3042dbb62af1a4f678409baf28f656488134ae98fdde6a WatchSource:0}: Error finding container 606693a8796bc424dd3042dbb62af1a4f678409baf28f656488134ae98fdde6a: Status 404 returned error can't find the container with id 606693a8796bc424dd3042dbb62af1a4f678409baf28f656488134ae98fdde6a
Oct 30 18:57:56.050342 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:56.050321    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"]
Oct 30 18:57:56.052649 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:56.052619    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:57:56.052754 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:56.052683    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:57:56.052754 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:56.052716    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:57:56.691132 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:56.691094    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74" event={"ID":"7af3a685-9810-41d3-9ba3-97e510ce2d96","Type":"ContainerStarted","Data":"02c0211a7c28830e71815e9ce3e0c7d4d8a8b453c08bdb2ce0eac3cad3a7c8ea"}
Oct 30 18:57:56.691323 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:56.691159    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74" event={"ID":"7af3a685-9810-41d3-9ba3-97e510ce2d96","Type":"ContainerStarted","Data":"606693a8796bc424dd3042dbb62af1a4f678409baf28f656488134ae98fdde6a"}
Oct 30 18:57:56.692458 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:56.692435    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b44d0e1-86vs5"
Oct 30 18:57:56.692595 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:56.692436    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b44d0e1-86vs5" event={"ID":"ca973523-e0ce-42d4-9d1e-bd0eab35c776","Type":"ContainerDied","Data":"e5113bea08aff8a9dce197bcfdaf7bcbeeba41eb78ab7d7c7786d0e5d56aa3ed"}
Oct 30 18:57:56.692595 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:56.692560    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5113bea08aff8a9dce197bcfdaf7bcbeeba41eb78ab7d7c7786d0e5d56aa3ed"
Oct 30 18:57:56.951794 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:56.951745    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74" podStartSLOduration=1.95173106 podStartE2EDuration="1.95173106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:56.894336338 +0000 UTC m=+1278.171386779" watchObservedRunningTime="2026-10-30 18:57:56.95173106 +0000 UTC m=+1278.228781498"
Oct 30 18:57:57.583076 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:57.583038    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:57:57.584559 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:57.584534    4924 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 18:57:57.584680 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:57.584580    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74" podUID="7af3a685-9810-41d3-9ba3-97e510ce2d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 18:57:58.583811 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:58.583776    4924 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 18:57:58.584246 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:58.583842    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74" podUID="7af3a685-9810-41d3-9ba3-97e510ce2d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 18:57:59.583604 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:59.583572    4924 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 18:57:59.583774 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:57:59.583638    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74" podUID="7af3a685-9810-41d3-9ba3-97e510ce2d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 18:58:00.329496 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:00.329441    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:58:00.528536 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:00.528498    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:58:00.583111 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:00.583003    4924 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 18:58:00.583111 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:00.583079    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74" podUID="7af3a685-9810-41d3-9ba3-97e510ce2d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 18:58:01.583175 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:01.583141    4924 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 18:58:01.583563 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:01.583202    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74" podUID="7af3a685-9810-41d3-9ba3-97e510ce2d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 18:58:02.583618 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:02.583568    4924 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 18:58:02.584158 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:02.583655    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74" podUID="7af3a685-9810-41d3-9ba3-97e510ce2d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 18:58:03.164173 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:03.164087    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:58:03.355587 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:03.355547    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:58:03.587981 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:03.587903    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:58:03.588374 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:03.588354    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:58:03.589120 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:03.589103    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 18:58:05.307119 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:05.307085    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:58:10.282644 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.282612    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"]
Oct 30 18:58:10.283266 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.283245    4924 cpu_manager.go:414] "containerMap: removing container" podUID="ca973523-e0ce-42d4-9d1e-bd0eab35c776" containerName="revoke-keys"
Oct 30 18:58:10.283266 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.283265    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca973523-e0ce-42d4-9d1e-bd0eab35c776" containerName="revoke-keys"
Oct 30 18:58:10.283485 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.283471    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca973523-e0ce-42d4-9d1e-bd0eab35c776" containerName="revoke-keys"
Oct 30 18:58:10.287237 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.287220    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.288647 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.288624    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:58:10.304492 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.304473    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkwdn\" (UniqueName: \"kubernetes.io/projected/0e31d430-f842-4ba1-9890-6b1000521b5d-kube-api-access-qkwdn\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.304589 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.304513    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-data\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.304631 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.304623    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-workload-certs\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.304670 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.304655    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-envoy\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.304740 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.304721    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-credential-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.304831 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.304816    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-podinfo\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.304865 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.304851    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-token\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.304952 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.304938    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-workload-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.304992 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.304977    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e31d430-f842-4ba1-9890-6b1000521b5d-istiod-ca-cert\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.380856 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.380814    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"]
Oct 30 18:58:10.405764 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.405737    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-workload-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.405909 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.405778    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e31d430-f842-4ba1-9890-6b1000521b5d-istiod-ca-cert\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.405909 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.405804    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkwdn\" (UniqueName: \"kubernetes.io/projected/0e31d430-f842-4ba1-9890-6b1000521b5d-kube-api-access-qkwdn\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.405909 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.405848    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-data\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.406078 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.406002    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-workload-certs\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.406134 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.406088    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-envoy\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.406134 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.406120    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-credential-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.406242 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.406187    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-workload-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.406242 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.406226    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-podinfo\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.406347 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.406260    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-data\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.406347 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.406274    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-token\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.406449 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.406389    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-workload-certs\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.406503 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.406483    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-credential-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.406642 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.406617    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e31d430-f842-4ba1-9890-6b1000521b5d-istiod-ca-cert\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.408504 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.408482    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-envoy\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.408698 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.408654    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-podinfo\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.481293 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.481267    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-token\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.481293 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.481280    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkwdn\" (UniqueName: \"kubernetes.io/projected/0e31d430-f842-4ba1-9890-6b1000521b5d-kube-api-access-qkwdn\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:10.508442 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.508421    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:58:10.601093 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:10.601054    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:11.021201 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:58:11.021144    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e31d430_f842_4ba1_9890_6b1000521b5d.slice/crio-139bd20044a901cf33b9da08a2ac7193dbea46d720715301adcea54f62b92b59 WatchSource:0}: Error finding container 139bd20044a901cf33b9da08a2ac7193dbea46d720715301adcea54f62b92b59: Status 404 returned error can't find the container with id 139bd20044a901cf33b9da08a2ac7193dbea46d720715301adcea54f62b92b59
Oct 30 18:58:11.021801 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:11.021775    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"]
Oct 30 18:58:11.023850 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:11.023818    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:11.023939 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:11.023879    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:11.023939 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:11.023905    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:11.837344 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:11.837305    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5" event={"ID":"0e31d430-f842-4ba1-9890-6b1000521b5d","Type":"ContainerStarted","Data":"d98742fb6ef83fe326b8650726d2dce31df92d65e17f37d4920c5e85d7b4f802"}
Oct 30 18:58:11.837827 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:11.837392    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5" event={"ID":"0e31d430-f842-4ba1-9890-6b1000521b5d","Type":"ContainerStarted","Data":"139bd20044a901cf33b9da08a2ac7193dbea46d720715301adcea54f62b92b59"}
Oct 30 18:58:11.998854 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:11.998804    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5" podStartSLOduration=1.998791115 podStartE2EDuration="1.998791115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:11.99594495 +0000 UTC m=+1293.272995394" watchObservedRunningTime="2026-10-30 18:58:11.998791115 +0000 UTC m=+1293.275841559"
Oct 30 18:58:12.614089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:12.602109    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:12.622485 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:12.622447    4924 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 18:58:12.622773 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:12.622744    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5" podUID="0e31d430-f842-4ba1-9890-6b1000521b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 18:58:13.601492 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:13.601461    4924 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 18:58:13.601901 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:13.601517    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5" podUID="0e31d430-f842-4ba1-9890-6b1000521b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 18:58:14.619401 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:14.602127    4924 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 18:58:14.627216 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:14.625151    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5" podUID="0e31d430-f842-4ba1-9890-6b1000521b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 18:58:15.238264 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:15.238230    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:58:15.601773 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:15.601742    4924 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 18:58:15.601933 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:15.601799    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5" podUID="0e31d430-f842-4ba1-9890-6b1000521b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 18:58:15.738708 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:58:15.738661    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:58:15.797890 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:15.797855    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe40c14f-kfprk"]
Oct 30 18:58:15.824384 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:15.824351    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe40c14f-kfprk"
Oct 30 18:58:15.854904 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:15.854821    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe40c14f-kfprk"]
Oct 30 18:58:15.880596 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:15.880561    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkfth\" (UniqueName: \"kubernetes.io/projected/5ae49714-1974-4577-b226-b6a0189673ed-kube-api-access-zkfth\") pod \"maas-api-revoke-sub-fe40c14f-kfprk\" (UID: \"5ae49714-1974-4577-b226-b6a0189673ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe40c14f-kfprk"
Oct 30 18:58:15.880755 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:15.880641    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae49714-1974-4577-b226-b6a0189673ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe40c14f-kfprk\" (UID: \"5ae49714-1974-4577-b226-b6a0189673ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe40c14f-kfprk"
Oct 30 18:58:15.880966 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:15.880940    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae49714-1974-4577-b226-b6a0189673ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe40c14f-kfprk\" (UID: \"5ae49714-1974-4577-b226-b6a0189673ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe40c14f-kfprk"
Oct 30 18:58:15.896362 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:15.896235    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:58:15.896362 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:15.896305    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 18:58:15.896362 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:15.896353    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:58:15.926085 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:15.925276    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:58:15.982017 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:15.981979    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae49714-1974-4577-b226-b6a0189673ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe40c14f-kfprk\" (UID: \"5ae49714-1974-4577-b226-b6a0189673ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe40c14f-kfprk"
Oct 30 18:58:15.982193 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:15.982076    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkfth\" (UniqueName: \"kubernetes.io/projected/5ae49714-1974-4577-b226-b6a0189673ed-kube-api-access-zkfth\") pod \"maas-api-revoke-sub-fe40c14f-kfprk\" (UID: \"5ae49714-1974-4577-b226-b6a0189673ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe40c14f-kfprk"
Oct 30 18:58:15.982193 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:15.982152    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae49714-1974-4577-b226-b6a0189673ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe40c14f-kfprk\" (UID: \"5ae49714-1974-4577-b226-b6a0189673ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe40c14f-kfprk"
Oct 30 18:58:15.982772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:15.982747    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae49714-1974-4577-b226-b6a0189673ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe40c14f-kfprk\" (UID: \"5ae49714-1974-4577-b226-b6a0189673ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe40c14f-kfprk"
Oct 30 18:58:15.984502 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:15.984476    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae49714-1974-4577-b226-b6a0189673ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe40c14f-kfprk\" (UID: \"5ae49714-1974-4577-b226-b6a0189673ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe40c14f-kfprk"
Oct 30 18:58:16.043139 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:16.043103    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkfth\" (UniqueName: \"kubernetes.io/projected/5ae49714-1974-4577-b226-b6a0189673ed-kube-api-access-zkfth\") pod \"maas-api-revoke-sub-fe40c14f-kfprk\" (UID: \"5ae49714-1974-4577-b226-b6a0189673ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe40c14f-kfprk"
Oct 30 18:58:16.145088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:16.144992    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe40c14f-kfprk"
Oct 30 18:58:16.542752 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:16.542724    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe40c14f-kfprk"]
Oct 30 18:58:16.634014 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:16.633557    4924 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 18:58:16.634014 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:16.633628    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5" podUID="0e31d430-f842-4ba1-9890-6b1000521b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 18:58:16.879603 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:16.879561    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe40c14f-kfprk" event={"ID":"5ae49714-1974-4577-b226-b6a0189673ed","Type":"ContainerStarted","Data":"427c2249db499b32bd990aac433c13e948c85c8d892c977c5708964d4c0101b7"}
Oct 30 18:58:17.601692 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:17.601659    4924 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 18:58:17.601899 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:17.601720    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5" podUID="0e31d430-f842-4ba1-9890-6b1000521b5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 18:58:17.888622 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:17.888552    4924 generic.go:356] "Generic (PLEG): container finished" podID="5ae49714-1974-4577-b226-b6a0189673ed" containerID="ad7003f179408de6a8ff0905bd0ca9ff7444199f027384abd9572077627091a7" exitCode=0
Oct 30 18:58:17.888969 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:17.888773    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe40c14f-kfprk" event={"ID":"5ae49714-1974-4577-b226-b6a0189673ed","Type":"ContainerDied","Data":"ad7003f179408de6a8ff0905bd0ca9ff7444199f027384abd9572077627091a7"}
Oct 30 18:58:18.962358 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:18.962314    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:18.963583 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:18.963299    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:18.964238 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:18.964164    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 18:58:19.067670 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:19.067648    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe40c14f-kfprk"
Oct 30 18:58:19.147466 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:19.147396    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ae49714-1974-4577-b226-b6a0189673ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae49714-1974-4577-b226-b6a0189673ed-maas-api-openshift-ca\") pod \"5ae49714-1974-4577-b226-b6a0189673ed\" (UID: \"5ae49714-1974-4577-b226-b6a0189673ed\") "
Oct 30 18:58:19.147630 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:19.147493    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae49714-1974-4577-b226-b6a0189673ed-kube-api-access-zkfth\" (UniqueName: \"kubernetes.io/projected/5ae49714-1974-4577-b226-b6a0189673ed-kube-api-access-zkfth\") pod \"5ae49714-1974-4577-b226-b6a0189673ed\" (UID: \"5ae49714-1974-4577-b226-b6a0189673ed\") "
Oct 30 18:58:19.147630 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:19.147542    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ae49714-1974-4577-b226-b6a0189673ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae49714-1974-4577-b226-b6a0189673ed-maas-api-xks-ca\") pod \"5ae49714-1974-4577-b226-b6a0189673ed\" (UID: \"5ae49714-1974-4577-b226-b6a0189673ed\") "
Oct 30 18:58:19.147805 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:19.147780    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae49714-1974-4577-b226-b6a0189673ed-maas-api-openshift-ca" pod "5ae49714-1974-4577-b226-b6a0189673ed" (UID: "5ae49714-1974-4577-b226-b6a0189673ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:19.147964 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:19.147945    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ae49714-1974-4577-b226-b6a0189673ed-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:19.149638 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:19.149610    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ae49714-1974-4577-b226-b6a0189673ed-maas-api-xks-ca" pod "5ae49714-1974-4577-b226-b6a0189673ed" (UID: "5ae49714-1974-4577-b226-b6a0189673ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:58:19.149749 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:19.149698    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae49714-1974-4577-b226-b6a0189673ed-kube-api-access-zkfth" pod "5ae49714-1974-4577-b226-b6a0189673ed" (UID: "5ae49714-1974-4577-b226-b6a0189673ed"). InnerVolumeSpecName "kube-api-access-zkfth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:19.249514 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:19.249483    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkfth\" (UniqueName: \"kubernetes.io/projected/5ae49714-1974-4577-b226-b6a0189673ed-kube-api-access-zkfth\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:19.249514 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:19.249512    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ae49714-1974-4577-b226-b6a0189673ed-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:19.905182 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:19.905108    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe40c14f-kfprk"
Oct 30 18:58:19.905182 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:19.905106    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe40c14f-kfprk" event={"ID":"5ae49714-1974-4577-b226-b6a0189673ed","Type":"ContainerDied","Data":"427c2249db499b32bd990aac433c13e948c85c8d892c977c5708964d4c0101b7"}
Oct 30 18:58:19.905415 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:19.905193    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="427c2249db499b32bd990aac433c13e948c85c8d892c977c5708964d4c0101b7"
Oct 30 18:58:20.327900 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:20.327789    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:58:24.223033 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:24.223002    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:58:25.072219 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:25.072187    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b80ef4f4-dkwc6"]
Oct 30 18:58:25.119025 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:25.118993    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b80ef4f4-dkwc6"]
Oct 30 18:58:25.445635 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:25.445600    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:58:26.835100 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:26.835050    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b956f39-ff10-4578-a5a8-cd17cb0e71fe" path="/var/lib/kubelet/pods/3b956f39-ff10-4578-a5a8-cd17cb0e71fe/volumes"
Oct 30 18:58:30.004167 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:30.004133    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034fccb1-rgxw7"]
Oct 30 18:58:30.005044 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:30.005019    4924 cpu_manager.go:414] "containerMap: removing container" podUID="5ae49714-1974-4577-b226-b6a0189673ed" containerName="revoke-keys"
Oct 30 18:58:30.005044 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:30.005043    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ae49714-1974-4577-b226-b6a0189673ed" containerName="revoke-keys"
Oct 30 18:58:30.005256 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:30.005239    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ae49714-1974-4577-b226-b6a0189673ed" containerName="revoke-keys"
Oct 30 18:58:30.008905 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:30.008883    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034fccb1-rgxw7"
Oct 30 18:58:30.059440 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:30.059417    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:58:30.059541 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:30.059520    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 18:58:30.082416 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:30.082389    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/052fcea7-50f4-444b-ad19-6c7346019e00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-034fccb1-rgxw7\" (UID: \"052fcea7-50f4-444b-ad19-6c7346019e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034fccb1-rgxw7"
Oct 30 18:58:30.082656 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:30.082633    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxr79\" (UniqueName: \"kubernetes.io/projected/052fcea7-50f4-444b-ad19-6c7346019e00-kube-api-access-wxr79\") pod \"maas-api-revoke-sub-034fccb1-rgxw7\" (UID: \"052fcea7-50f4-444b-ad19-6c7346019e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034fccb1-rgxw7"
Oct 30 18:58:30.082824 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:30.082792    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/052fcea7-50f4-444b-ad19-6c7346019e00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-034fccb1-rgxw7\" (UID: \"052fcea7-50f4-444b-ad19-6c7346019e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034fccb1-rgxw7"
Oct 30 18:58:30.106134 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:30.106108    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:58:30.106289 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:30.106224    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:58:30.115932 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:30.115903    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034fccb1-rgxw7"]
Oct 30 18:58:30.183858 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:30.183826    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/052fcea7-50f4-444b-ad19-6c7346019e00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-034fccb1-rgxw7\" (UID: \"052fcea7-50f4-444b-ad19-6c7346019e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034fccb1-rgxw7"
Oct 30 18:58:30.184020 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:30.183881    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/052fcea7-50f4-444b-ad19-6c7346019e00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-034fccb1-rgxw7\" (UID: \"052fcea7-50f4-444b-ad19-6c7346019e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034fccb1-rgxw7"
Oct 30 18:58:30.184020 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:30.184016    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxr79\" (UniqueName: \"kubernetes.io/projected/052fcea7-50f4-444b-ad19-6c7346019e00-kube-api-access-wxr79\") pod \"maas-api-revoke-sub-034fccb1-rgxw7\" (UID: \"052fcea7-50f4-444b-ad19-6c7346019e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034fccb1-rgxw7"
Oct 30 18:58:30.184504 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:30.184481    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/052fcea7-50f4-444b-ad19-6c7346019e00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-034fccb1-rgxw7\" (UID: \"052fcea7-50f4-444b-ad19-6c7346019e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034fccb1-rgxw7"
Oct 30 18:58:30.186348 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:30.186320    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/052fcea7-50f4-444b-ad19-6c7346019e00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-034fccb1-rgxw7\" (UID: \"052fcea7-50f4-444b-ad19-6c7346019e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034fccb1-rgxw7"
Oct 30 18:58:30.238523 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:30.238498    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxr79\" (UniqueName: \"kubernetes.io/projected/052fcea7-50f4-444b-ad19-6c7346019e00-kube-api-access-wxr79\") pod \"maas-api-revoke-sub-034fccb1-rgxw7\" (UID: \"052fcea7-50f4-444b-ad19-6c7346019e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034fccb1-rgxw7"
Oct 30 18:58:30.321764 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:30.321683    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034fccb1-rgxw7"
Oct 30 18:58:30.476373 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:30.476333    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:58:30.700036 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:30.700008    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034fccb1-rgxw7"]
Oct 30 18:58:30.984245 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:30.984171    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034fccb1-rgxw7" event={"ID":"052fcea7-50f4-444b-ad19-6c7346019e00","Type":"ContainerStarted","Data":"8edb58f44d41db9ae4609308a7e817d99453b9c2745d7313d7e90b855917e4b9"}
Oct 30 18:58:31.993319 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:31.993288    4924 generic.go:356] "Generic (PLEG): container finished" podID="052fcea7-50f4-444b-ad19-6c7346019e00" containerID="72069bfc22827e46458e93a452117d1e33c4880f9cb37ea0a08d3da93d5e0f68" exitCode=0
Oct 30 18:58:31.993722 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:31.993378    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034fccb1-rgxw7" event={"ID":"052fcea7-50f4-444b-ad19-6c7346019e00","Type":"ContainerDied","Data":"72069bfc22827e46458e93a452117d1e33c4880f9cb37ea0a08d3da93d5e0f68"}
Oct 30 18:58:33.148183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:33.148160    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034fccb1-rgxw7"
Oct 30 18:58:33.225311 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:33.225286    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/052fcea7-50f4-444b-ad19-6c7346019e00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/052fcea7-50f4-444b-ad19-6c7346019e00-maas-api-openshift-ca\") pod \"052fcea7-50f4-444b-ad19-6c7346019e00\" (UID: \"052fcea7-50f4-444b-ad19-6c7346019e00\") "
Oct 30 18:58:33.225433 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:33.225357    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/052fcea7-50f4-444b-ad19-6c7346019e00-kube-api-access-wxr79\" (UniqueName: \"kubernetes.io/projected/052fcea7-50f4-444b-ad19-6c7346019e00-kube-api-access-wxr79\") pod \"052fcea7-50f4-444b-ad19-6c7346019e00\" (UID: \"052fcea7-50f4-444b-ad19-6c7346019e00\") "
Oct 30 18:58:33.225510 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:33.225493    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/052fcea7-50f4-444b-ad19-6c7346019e00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/052fcea7-50f4-444b-ad19-6c7346019e00-maas-api-xks-ca\") pod \"052fcea7-50f4-444b-ad19-6c7346019e00\" (UID: \"052fcea7-50f4-444b-ad19-6c7346019e00\") "
Oct 30 18:58:33.225678 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:33.225658    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/052fcea7-50f4-444b-ad19-6c7346019e00-maas-api-openshift-ca" pod "052fcea7-50f4-444b-ad19-6c7346019e00" (UID: "052fcea7-50f4-444b-ad19-6c7346019e00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:33.226086 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:33.226042    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/052fcea7-50f4-444b-ad19-6c7346019e00-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:33.227561 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:33.227533    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052fcea7-50f4-444b-ad19-6c7346019e00-kube-api-access-wxr79" pod "052fcea7-50f4-444b-ad19-6c7346019e00" (UID: "052fcea7-50f4-444b-ad19-6c7346019e00"). InnerVolumeSpecName "kube-api-access-wxr79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:33.227650 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:33.227582    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/052fcea7-50f4-444b-ad19-6c7346019e00-maas-api-xks-ca" pod "052fcea7-50f4-444b-ad19-6c7346019e00" (UID: "052fcea7-50f4-444b-ad19-6c7346019e00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:58:33.327712 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:33.327630    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxr79\" (UniqueName: \"kubernetes.io/projected/052fcea7-50f4-444b-ad19-6c7346019e00-kube-api-access-wxr79\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:33.327712 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:33.327663    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/052fcea7-50f4-444b-ad19-6c7346019e00-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:34.012031 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:34.011991    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034fccb1-rgxw7" event={"ID":"052fcea7-50f4-444b-ad19-6c7346019e00","Type":"ContainerDied","Data":"8edb58f44d41db9ae4609308a7e817d99453b9c2745d7313d7e90b855917e4b9"}
Oct 30 18:58:34.012031 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:34.012010    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034fccb1-rgxw7"
Oct 30 18:58:34.012249 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:34.012045    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8edb58f44d41db9ae4609308a7e817d99453b9c2745d7313d7e90b855917e4b9"
Oct 30 18:58:35.324085 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:35.323925    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:58:35.605167 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:35.605129    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:58:40.326252 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.326222    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"]
Oct 30 18:58:40.326813 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.326796    4924 cpu_manager.go:414] "containerMap: removing container" podUID="052fcea7-50f4-444b-ad19-6c7346019e00" containerName="revoke-keys"
Oct 30 18:58:40.326878 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.326816    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="052fcea7-50f4-444b-ad19-6c7346019e00" containerName="revoke-keys"
Oct 30 18:58:40.327049 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.327034    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="052fcea7-50f4-444b-ad19-6c7346019e00" containerName="revoke-keys"
Oct 30 18:58:40.331830 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.331813    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.415925 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.415884    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:58:40.418380 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.418350    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-istio-envoy\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.418706 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.418673    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-workload-certs\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.418827 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.418746    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4947eb13-56e8-4d54-b253-21913852aeb1-istio-token\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.418827 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.418785    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4947eb13-56e8-4d54-b253-21913852aeb1-istiod-ca-cert\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.420413 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.418828    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-workload-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.420413 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.418886    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4947eb13-56e8-4d54-b253-21913852aeb1-istio-podinfo\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.421316 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.421282    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-credential-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.421430 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.421362    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-istio-data\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.421430 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.421412    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvgn5\" (UniqueName: \"kubernetes.io/projected/4947eb13-56e8-4d54-b253-21913852aeb1-kube-api-access-bvgn5\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.463961 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.463925    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"]
Oct 30 18:58:40.523221 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.523187    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-workload-certs\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.523404 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.523239    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4947eb13-56e8-4d54-b253-21913852aeb1-istio-token\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.523404 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.523352    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4947eb13-56e8-4d54-b253-21913852aeb1-istiod-ca-cert\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.523404 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.523398    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-workload-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.523583 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.523458    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4947eb13-56e8-4d54-b253-21913852aeb1-istio-podinfo\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.523583 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.523572    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-credential-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.523694 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.523636    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-istio-data\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.523694 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.523644    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-workload-certs\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.523694 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.523679    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvgn5\" (UniqueName: \"kubernetes.io/projected/4947eb13-56e8-4d54-b253-21913852aeb1-kube-api-access-bvgn5\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.523829 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.523708    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-workload-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.523829 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.523819    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-istio-envoy\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.523971 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.523940    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-credential-socket\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.524119 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.523979    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-istio-data\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.524204 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.524176    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4947eb13-56e8-4d54-b253-21913852aeb1-istiod-ca-cert\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.525924 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.525903    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-istio-envoy\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.526018 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.525969    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4947eb13-56e8-4d54-b253-21913852aeb1-istio-podinfo\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.577830 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.577768    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4947eb13-56e8-4d54-b253-21913852aeb1-istio-token\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.578023 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.578002    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvgn5\" (UniqueName: \"kubernetes.io/projected/4947eb13-56e8-4d54-b253-21913852aeb1-kube-api-access-bvgn5\") pod \"e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") " pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.644960 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.644933    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:40.750917 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.750883    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82aee4e5-fnt28"]
Oct 30 18:58:40.757160 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.757140    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82aee4e5-fnt28"
Oct 30 18:58:40.825009 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.824984    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:58:40.825183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.825135    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:58:40.825183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.825135    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 18:58:40.827992 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.827931    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtj52\" (UniqueName: \"kubernetes.io/projected/add5882b-25a6-4736-b414-9ebea55362e0-kube-api-access-gtj52\") pod \"maas-api-revoke-sub-82aee4e5-fnt28\" (UID: \"add5882b-25a6-4736-b414-9ebea55362e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82aee4e5-fnt28"
Oct 30 18:58:40.827992 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.827970    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add5882b-25a6-4736-b414-9ebea55362e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82aee4e5-fnt28\" (UID: \"add5882b-25a6-4736-b414-9ebea55362e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82aee4e5-fnt28"
Oct 30 18:58:40.828145 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.828127    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add5882b-25a6-4736-b414-9ebea55362e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82aee4e5-fnt28\" (UID: \"add5882b-25a6-4736-b414-9ebea55362e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82aee4e5-fnt28"
Oct 30 18:58:40.828475 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.828444    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:58:40.837132 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.837103    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82aee4e5-fnt28"]
Oct 30 18:58:40.929873 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.929843    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtj52\" (UniqueName: \"kubernetes.io/projected/add5882b-25a6-4736-b414-9ebea55362e0-kube-api-access-gtj52\") pod \"maas-api-revoke-sub-82aee4e5-fnt28\" (UID: \"add5882b-25a6-4736-b414-9ebea55362e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82aee4e5-fnt28"
Oct 30 18:58:40.930012 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.929893    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add5882b-25a6-4736-b414-9ebea55362e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82aee4e5-fnt28\" (UID: \"add5882b-25a6-4736-b414-9ebea55362e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82aee4e5-fnt28"
Oct 30 18:58:40.930055 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.930040    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add5882b-25a6-4736-b414-9ebea55362e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82aee4e5-fnt28\" (UID: \"add5882b-25a6-4736-b414-9ebea55362e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82aee4e5-fnt28"
Oct 30 18:58:40.930713 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.930693    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add5882b-25a6-4736-b414-9ebea55362e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82aee4e5-fnt28\" (UID: \"add5882b-25a6-4736-b414-9ebea55362e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82aee4e5-fnt28"
Oct 30 18:58:40.932269 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.932248    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add5882b-25a6-4736-b414-9ebea55362e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82aee4e5-fnt28\" (UID: \"add5882b-25a6-4736-b414-9ebea55362e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82aee4e5-fnt28"
Oct 30 18:58:40.984831 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:40.984812    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtj52\" (UniqueName: \"kubernetes.io/projected/add5882b-25a6-4736-b414-9ebea55362e0-kube-api-access-gtj52\") pod \"maas-api-revoke-sub-82aee4e5-fnt28\" (UID: \"add5882b-25a6-4736-b414-9ebea55362e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82aee4e5-fnt28"
Oct 30 18:58:41.071058 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:41.071038    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82aee4e5-fnt28"
Oct 30 18:58:41.074417 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:58:41.074387    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4947eb13_56e8_4d54_b253_21913852aeb1.slice/crio-08fe2b4acff8d25d7c973148d0f802e2334963dddeabdf1557e131a806ab9d9b WatchSource:0}: Error finding container 08fe2b4acff8d25d7c973148d0f802e2334963dddeabdf1557e131a806ab9d9b: Status 404 returned error can't find the container with id 08fe2b4acff8d25d7c973148d0f802e2334963dddeabdf1557e131a806ab9d9b
Oct 30 18:58:41.074833 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:41.074811    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"]
Oct 30 18:58:41.081924 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:41.081853    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:41.081999 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:41.081937    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:41.082038 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:41.081998    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:41.455145 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:41.455117    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82aee4e5-fnt28"]
Oct 30 18:58:41.455503 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:58:41.455403    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd5882b_25a6_4736_b414_9ebea55362e0.slice/crio-421b6160d5d99be2483a54792f4060d0fea829208f62bb5d2c329661afd69b96 WatchSource:0}: Error finding container 421b6160d5d99be2483a54792f4060d0fea829208f62bb5d2c329661afd69b96: Status 404 returned error can't find the container with id 421b6160d5d99be2483a54792f4060d0fea829208f62bb5d2c329661afd69b96
Oct 30 18:58:42.074191 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:42.074101    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl" event={"ID":"4947eb13-56e8-4d54-b253-21913852aeb1","Type":"ContainerStarted","Data":"16be12feb68fac8f7a9066d85e368b7d706ffe6d221c3583b9fc937cee169362"}
Oct 30 18:58:42.074191 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:42.074164    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl" event={"ID":"4947eb13-56e8-4d54-b253-21913852aeb1","Type":"ContainerStarted","Data":"08fe2b4acff8d25d7c973148d0f802e2334963dddeabdf1557e131a806ab9d9b"}
Oct 30 18:58:42.075805 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:42.075778    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82aee4e5-fnt28" event={"ID":"add5882b-25a6-4736-b414-9ebea55362e0","Type":"ContainerStarted","Data":"dfb41da98e66c3f09f405eaea91c44e26b048e525b130000f9c0bd10ff882df2"}
Oct 30 18:58:42.075936 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:42.075826    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82aee4e5-fnt28" event={"ID":"add5882b-25a6-4736-b414-9ebea55362e0","Type":"ContainerStarted","Data":"421b6160d5d99be2483a54792f4060d0fea829208f62bb5d2c329661afd69b96"}
Oct 30 18:58:42.190024 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:42.189971    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl" podStartSLOduration=2.189956983 podStartE2EDuration="2.189956983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:42.188318849 +0000 UTC m=+1323.465369295" watchObservedRunningTime="2026-10-30 18:58:42.189956983 +0000 UTC m=+1323.467007424"
Oct 30 18:58:42.314106 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:42.313623    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82aee4e5-fnt28" podStartSLOduration=2.3136 podStartE2EDuration="2.3136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:42.297606659 +0000 UTC m=+1323.574657103" watchObservedRunningTime="2026-10-30 18:58:42.3136 +0000 UTC m=+1323.590650448"
Oct 30 18:58:42.645140 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:42.645097    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:42.646766 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:42.646740    4924 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 18:58:42.646881 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:42.646788    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl" podUID="4947eb13-56e8-4d54-b253-21913852aeb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 18:58:43.085315 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:43.085235    4924 generic.go:356] "Generic (PLEG): container finished" podID="add5882b-25a6-4736-b414-9ebea55362e0" containerID="dfb41da98e66c3f09f405eaea91c44e26b048e525b130000f9c0bd10ff882df2" exitCode=0
Oct 30 18:58:43.085491 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:43.085318    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82aee4e5-fnt28" event={"ID":"add5882b-25a6-4736-b414-9ebea55362e0","Type":"ContainerDied","Data":"dfb41da98e66c3f09f405eaea91c44e26b048e525b130000f9c0bd10ff882df2"}
Oct 30 18:58:43.645398 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:43.645367    4924 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 18:58:43.645895 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:43.645865    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl" podUID="4947eb13-56e8-4d54-b253-21913852aeb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 18:58:44.298954 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:44.298928    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82aee4e5-fnt28"
Oct 30 18:58:44.376733 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:44.376703    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/add5882b-25a6-4736-b414-9ebea55362e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add5882b-25a6-4736-b414-9ebea55362e0-maas-api-openshift-ca\") pod \"add5882b-25a6-4736-b414-9ebea55362e0\" (UID: \"add5882b-25a6-4736-b414-9ebea55362e0\") "
Oct 30 18:58:44.376908 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:44.376883    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/add5882b-25a6-4736-b414-9ebea55362e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add5882b-25a6-4736-b414-9ebea55362e0-maas-api-xks-ca\") pod \"add5882b-25a6-4736-b414-9ebea55362e0\" (UID: \"add5882b-25a6-4736-b414-9ebea55362e0\") "
Oct 30 18:58:44.376975 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:44.376928    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add5882b-25a6-4736-b414-9ebea55362e0-kube-api-access-gtj52\" (UniqueName: \"kubernetes.io/projected/add5882b-25a6-4736-b414-9ebea55362e0-kube-api-access-gtj52\") pod \"add5882b-25a6-4736-b414-9ebea55362e0\" (UID: \"add5882b-25a6-4736-b414-9ebea55362e0\") "
Oct 30 18:58:44.377034 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:44.376970    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add5882b-25a6-4736-b414-9ebea55362e0-maas-api-openshift-ca" pod "add5882b-25a6-4736-b414-9ebea55362e0" (UID: "add5882b-25a6-4736-b414-9ebea55362e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:44.377454 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:44.377428    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add5882b-25a6-4736-b414-9ebea55362e0-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:44.379005 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:44.378985    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/add5882b-25a6-4736-b414-9ebea55362e0-maas-api-xks-ca" pod "add5882b-25a6-4736-b414-9ebea55362e0" (UID: "add5882b-25a6-4736-b414-9ebea55362e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:58:44.379115 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:44.379091    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add5882b-25a6-4736-b414-9ebea55362e0-kube-api-access-gtj52" pod "add5882b-25a6-4736-b414-9ebea55362e0" (UID: "add5882b-25a6-4736-b414-9ebea55362e0"). InnerVolumeSpecName "kube-api-access-gtj52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:44.479125 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:44.479092    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtj52\" (UniqueName: \"kubernetes.io/projected/add5882b-25a6-4736-b414-9ebea55362e0-kube-api-access-gtj52\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:44.479125 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:44.479122    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add5882b-25a6-4736-b414-9ebea55362e0-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:44.645995 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:44.645912    4924 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 18:58:44.645995 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:44.645963    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl" podUID="4947eb13-56e8-4d54-b253-21913852aeb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 18:58:45.100529 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:45.100504    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82aee4e5-fnt28"
Oct 30 18:58:45.100689 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:45.100538    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82aee4e5-fnt28" event={"ID":"add5882b-25a6-4736-b414-9ebea55362e0","Type":"ContainerDied","Data":"421b6160d5d99be2483a54792f4060d0fea829208f62bb5d2c329661afd69b96"}
Oct 30 18:58:45.100689 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:45.100587    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="421b6160d5d99be2483a54792f4060d0fea829208f62bb5d2c329661afd69b96"
Oct 30 18:58:45.342277 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:45.341929    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:58:45.645631 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:45.645595    4924 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 18:58:45.645804 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:45.645659    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl" podUID="4947eb13-56e8-4d54-b253-21913852aeb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 18:58:45.736443 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:58:45.736394    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:58:46.645573 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:46.645540    4924 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 18:58:46.645756 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:46.645605    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl" podUID="4947eb13-56e8-4d54-b253-21913852aeb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 18:58:47.645793 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:47.645754    4924 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 18:58:47.646230 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:47.645809    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl" podUID="4947eb13-56e8-4d54-b253-21913852aeb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 18:58:48.841783 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:48.841752    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:48.842706 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:48.842682    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:48.843581 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:48.843559    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 18:58:50.446949 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:50.446898    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:58:51.066916 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:51.066883    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4c5f6b6-pc8ww"]
Oct 30 18:58:51.113037 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:51.113006    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4c5f6b6-pc8ww"]
Oct 30 18:58:51.341684 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:51.341600    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5eaa0bd-lrwlw"]
Oct 30 18:58:51.342381 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:51.342358    4924 cpu_manager.go:414] "containerMap: removing container" podUID="add5882b-25a6-4736-b414-9ebea55362e0" containerName="revoke-keys"
Oct 30 18:58:51.342381 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:51.342380    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add5882b-25a6-4736-b414-9ebea55362e0" containerName="revoke-keys"
Oct 30 18:58:51.342577 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:51.342562    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="add5882b-25a6-4736-b414-9ebea55362e0" containerName="revoke-keys"
Oct 30 18:58:51.347320 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:51.347300    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5eaa0bd-lrwlw"
Oct 30 18:58:51.411670 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:51.411625    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:58:51.411670 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:51.411662    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:58:51.411884 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:51.411664    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:58:51.411884 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:51.411756    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 18:58:51.416556 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:51.416533    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5eaa0bd-lrwlw"]
Oct 30 18:58:51.492640 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:51.492610    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88059f5b-33a7-407d-8c56-5656fc90cace-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5eaa0bd-lrwlw\" (UID: \"88059f5b-33a7-407d-8c56-5656fc90cace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5eaa0bd-lrwlw"
Oct 30 18:58:51.493109 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:51.492820    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqjzr\" (UniqueName: \"kubernetes.io/projected/88059f5b-33a7-407d-8c56-5656fc90cace-kube-api-access-gqjzr\") pod \"maas-api-revoke-sub-c5eaa0bd-lrwlw\" (UID: \"88059f5b-33a7-407d-8c56-5656fc90cace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5eaa0bd-lrwlw"
Oct 30 18:58:51.493109 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:51.492999    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88059f5b-33a7-407d-8c56-5656fc90cace-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5eaa0bd-lrwlw\" (UID: \"88059f5b-33a7-407d-8c56-5656fc90cace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5eaa0bd-lrwlw"
Oct 30 18:58:51.594265 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:51.594198    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88059f5b-33a7-407d-8c56-5656fc90cace-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5eaa0bd-lrwlw\" (UID: \"88059f5b-33a7-407d-8c56-5656fc90cace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5eaa0bd-lrwlw"
Oct 30 18:58:51.594265 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:51.594257    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88059f5b-33a7-407d-8c56-5656fc90cace-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5eaa0bd-lrwlw\" (UID: \"88059f5b-33a7-407d-8c56-5656fc90cace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5eaa0bd-lrwlw"
Oct 30 18:58:51.594455 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:51.594365    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqjzr\" (UniqueName: \"kubernetes.io/projected/88059f5b-33a7-407d-8c56-5656fc90cace-kube-api-access-gqjzr\") pod \"maas-api-revoke-sub-c5eaa0bd-lrwlw\" (UID: \"88059f5b-33a7-407d-8c56-5656fc90cace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5eaa0bd-lrwlw"
Oct 30 18:58:51.594944 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:51.594921    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88059f5b-33a7-407d-8c56-5656fc90cace-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5eaa0bd-lrwlw\" (UID: \"88059f5b-33a7-407d-8c56-5656fc90cace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5eaa0bd-lrwlw"
Oct 30 18:58:51.596504 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:51.596485    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88059f5b-33a7-407d-8c56-5656fc90cace-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5eaa0bd-lrwlw\" (UID: \"88059f5b-33a7-407d-8c56-5656fc90cace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5eaa0bd-lrwlw"
Oct 30 18:58:51.667939 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:51.667903    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqjzr\" (UniqueName: \"kubernetes.io/projected/88059f5b-33a7-407d-8c56-5656fc90cace-kube-api-access-gqjzr\") pod \"maas-api-revoke-sub-c5eaa0bd-lrwlw\" (UID: \"88059f5b-33a7-407d-8c56-5656fc90cace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5eaa0bd-lrwlw"
Oct 30 18:58:51.961427 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:51.961398    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5eaa0bd-lrwlw"
Oct 30 18:58:52.332824 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:52.332787    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5eaa0bd-lrwlw"]
Oct 30 18:58:52.332969 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:58:52.332841    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88059f5b_33a7_407d_8c56_5656fc90cace.slice/crio-690a225717e21fa83c14be0071fe40994c3f519dbc16c7113cfb72f74f09869c WatchSource:0}: Error finding container 690a225717e21fa83c14be0071fe40994c3f519dbc16c7113cfb72f74f09869c: Status 404 returned error can't find the container with id 690a225717e21fa83c14be0071fe40994c3f519dbc16c7113cfb72f74f09869c
Oct 30 18:58:52.835337 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:52.835306    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbb247e2-a8a0-49b9-9f12-bdfd0f494b68" path="/var/lib/kubelet/pods/dbb247e2-a8a0-49b9-9f12-bdfd0f494b68/volumes"
Oct 30 18:58:53.186682 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:53.186642    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5eaa0bd-lrwlw" event={"ID":"88059f5b-33a7-407d-8c56-5656fc90cace","Type":"ContainerStarted","Data":"e17ab1ea861343ad922ed51fe6ede69e09a3a9a08f0e88ccbdeb9c032ac544e5"}
Oct 30 18:58:53.186842 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:53.186703    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5eaa0bd-lrwlw" event={"ID":"88059f5b-33a7-407d-8c56-5656fc90cace","Type":"ContainerStarted","Data":"690a225717e21fa83c14be0071fe40994c3f519dbc16c7113cfb72f74f09869c"}
Oct 30 18:58:53.309590 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:53.309543    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5eaa0bd-lrwlw" podStartSLOduration=2.309529531 podStartE2EDuration="2.309529531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:53.307599057 +0000 UTC m=+1334.584649700" watchObservedRunningTime="2026-10-30 18:58:53.309529531 +0000 UTC m=+1334.586579977"
Oct 30 18:58:54.195493 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:54.195462    4924 generic.go:356] "Generic (PLEG): container finished" podID="88059f5b-33a7-407d-8c56-5656fc90cace" containerID="e17ab1ea861343ad922ed51fe6ede69e09a3a9a08f0e88ccbdeb9c032ac544e5" exitCode=0
Oct 30 18:58:54.195912 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:54.195547    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5eaa0bd-lrwlw" event={"ID":"88059f5b-33a7-407d-8c56-5656fc90cace","Type":"ContainerDied","Data":"e17ab1ea861343ad922ed51fe6ede69e09a3a9a08f0e88ccbdeb9c032ac544e5"}
Oct 30 18:58:55.331634 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.331603    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"]
Oct 30 18:58:55.336292 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.336262    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:58:55.361822 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.361785    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5eaa0bd-lrwlw"
Oct 30 18:58:55.390813 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.390777    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"]
Oct 30 18:58:55.432791 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.432759    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f80ac\"/\"distinct-2-w1-5f80ac-kserve-self-signed-certs\""
Oct 30 18:58:55.437225 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.437203    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88059f5b-33a7-407d-8c56-5656fc90cace-kube-api-access-gqjzr\" (UniqueName: \"kubernetes.io/projected/88059f5b-33a7-407d-8c56-5656fc90cace-kube-api-access-gqjzr\") pod \"88059f5b-33a7-407d-8c56-5656fc90cace\" (UID: \"88059f5b-33a7-407d-8c56-5656fc90cace\") "
Oct 30 18:58:55.437344 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.437315    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88059f5b-33a7-407d-8c56-5656fc90cace-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88059f5b-33a7-407d-8c56-5656fc90cace-maas-api-xks-ca\") pod \"88059f5b-33a7-407d-8c56-5656fc90cace\" (UID: \"88059f5b-33a7-407d-8c56-5656fc90cace\") "
Oct 30 18:58:55.437407 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.437396    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88059f5b-33a7-407d-8c56-5656fc90cace-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88059f5b-33a7-407d-8c56-5656fc90cace-maas-api-openshift-ca\") pod \"88059f5b-33a7-407d-8c56-5656fc90cace\" (UID: \"88059f5b-33a7-407d-8c56-5656fc90cace\") "
Oct 30 18:58:55.437629 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.437604    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-model-cache\") pod \"distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6\" (UID: \"eb82dd7d-e682-42a2-924e-a84f21172fc8\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:58:55.437748 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.437682    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdcsm\" (UniqueName: \"kubernetes.io/projected/eb82dd7d-e682-42a2-924e-a84f21172fc8-kube-api-access-wdcsm\") pod \"distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6\" (UID: \"eb82dd7d-e682-42a2-924e-a84f21172fc8\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:58:55.437817 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.437744    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88059f5b-33a7-407d-8c56-5656fc90cace-maas-api-openshift-ca" pod "88059f5b-33a7-407d-8c56-5656fc90cace" (UID: "88059f5b-33a7-407d-8c56-5656fc90cace"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:58:55.437878 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.437817    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-tmp-dir\") pod \"distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6\" (UID: \"eb82dd7d-e682-42a2-924e-a84f21172fc8\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:58:55.437924 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.437881    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-home\") pod \"distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6\" (UID: \"eb82dd7d-e682-42a2-924e-a84f21172fc8\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:58:55.437924 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.437907    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-dshm\") pod \"distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6\" (UID: \"eb82dd7d-e682-42a2-924e-a84f21172fc8\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:58:55.438146 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.438125    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb82dd7d-e682-42a2-924e-a84f21172fc8-tls-certs\") pod \"distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6\" (UID: \"eb82dd7d-e682-42a2-924e-a84f21172fc8\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:58:55.438290 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.438247    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88059f5b-33a7-407d-8c56-5656fc90cace-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:55.439465 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.439436    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88059f5b-33a7-407d-8c56-5656fc90cace-kube-api-access-gqjzr" pod "88059f5b-33a7-407d-8c56-5656fc90cace" (UID: "88059f5b-33a7-407d-8c56-5656fc90cace"). InnerVolumeSpecName "kube-api-access-gqjzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:55.439595 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.439527    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88059f5b-33a7-407d-8c56-5656fc90cace-maas-api-xks-ca" pod "88059f5b-33a7-407d-8c56-5656fc90cace" (UID: "88059f5b-33a7-407d-8c56-5656fc90cace"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:58:55.444389 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.444366    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"]
Oct 30 18:58:55.445133 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.445115    4924 cpu_manager.go:414] "containerMap: removing container" podUID="88059f5b-33a7-407d-8c56-5656fc90cace" containerName="revoke-keys"
Oct 30 18:58:55.445247 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.445134    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88059f5b-33a7-407d-8c56-5656fc90cace" containerName="revoke-keys"
Oct 30 18:58:55.445302 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.445263    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="88059f5b-33a7-407d-8c56-5656fc90cace" containerName="revoke-keys"
Oct 30 18:58:55.449129 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.449114    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.503039 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.503011    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"]
Oct 30 18:58:55.508346 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.508318    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"]
Oct 30 18:58:55.508474 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.508459    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.513218 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.513190    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:58:55.540618 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.540390    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-model-cache\") pod \"distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6\" (UID: \"eb82dd7d-e682-42a2-924e-a84f21172fc8\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:58:55.540618 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.540464    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdcsm\" (UniqueName: \"kubernetes.io/projected/eb82dd7d-e682-42a2-924e-a84f21172fc8-kube-api-access-wdcsm\") pod \"distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6\" (UID: \"eb82dd7d-e682-42a2-924e-a84f21172fc8\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:58:55.540618 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.540521    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-tmp-dir\") pod \"distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6\" (UID: \"eb82dd7d-e682-42a2-924e-a84f21172fc8\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:58:55.540618 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.540575    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-home\") pod \"distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6\" (UID: \"eb82dd7d-e682-42a2-924e-a84f21172fc8\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:58:55.540618 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.540611    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-dshm\") pod \"distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6\" (UID: \"eb82dd7d-e682-42a2-924e-a84f21172fc8\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:58:55.541011 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.540816    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb82dd7d-e682-42a2-924e-a84f21172fc8-tls-certs\") pod \"distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6\" (UID: \"eb82dd7d-e682-42a2-924e-a84f21172fc8\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:58:55.541011 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.540933    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqjzr\" (UniqueName: \"kubernetes.io/projected/88059f5b-33a7-407d-8c56-5656fc90cace-kube-api-access-gqjzr\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:55.541011 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.540950    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88059f5b-33a7-407d-8c56-5656fc90cace-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:55.541966 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.541936    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-home\") pod \"distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6\" (UID: \"eb82dd7d-e682-42a2-924e-a84f21172fc8\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:58:55.554724 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.544671    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-model-cache\") pod \"distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6\" (UID: \"eb82dd7d-e682-42a2-924e-a84f21172fc8\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:58:55.554724 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.545043    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-tmp-dir\") pod \"distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6\" (UID: \"eb82dd7d-e682-42a2-924e-a84f21172fc8\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:58:55.554724 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.554604    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb82dd7d-e682-42a2-924e-a84f21172fc8-tls-certs\") pod \"distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6\" (UID: \"eb82dd7d-e682-42a2-924e-a84f21172fc8\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:58:55.554932 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.554832    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-dshm\") pod \"distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6\" (UID: \"eb82dd7d-e682-42a2-924e-a84f21172fc8\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:58:55.572159 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.572125    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"]
Oct 30 18:58:55.602877 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.602842    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdcsm\" (UniqueName: \"kubernetes.io/projected/eb82dd7d-e682-42a2-924e-a84f21172fc8-kube-api-access-wdcsm\") pod \"distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6\" (UID: \"eb82dd7d-e682-42a2-924e-a84f21172fc8\") " pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:58:55.642508 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.642472    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-envoy\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.642688 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.642594    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jttf5\" (UniqueName: \"kubernetes.io/projected/3d89616c-b699-44df-983a-d58cad14b601-kube-api-access-jttf5\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.642688 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.642630    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-workload-certs\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.642778 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.642748    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-workload-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.642822 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.642774    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d89616c-b699-44df-983a-d58cad14b601-istio-podinfo\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.642901 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.642887    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-istio-envoy\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.642936 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.642930    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-istio-data\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.642968 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.642952    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d89616c-b699-44df-983a-d58cad14b601-istio-token\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.643000 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.642987    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d89616c-b699-44df-983a-d58cad14b601-istiod-ca-cert\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.643080 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.643046    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-credential-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.643229 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.643162    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l994m\" (UniqueName: \"kubernetes.io/projected/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-kube-api-access-l994m\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.643229 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.643201    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-podinfo\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.643322 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.643304    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-token\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.643362 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.643342    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-credential-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.643406 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.643363    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istiod-ca-cert\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.643455 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.643429    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-workload-certs\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.643509 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.643485    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-data\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.643563 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.643534    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-workload-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.672142 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.672119    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:58:55.690244 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.690222    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.745432    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-workload-certs\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.745554    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-data\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.745618    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-workload-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.745673    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-envoy\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.745725    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jttf5\" (UniqueName: \"kubernetes.io/projected/3d89616c-b699-44df-983a-d58cad14b601-kube-api-access-jttf5\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.745768    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-workload-certs\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.745818    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-workload-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.745845    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d89616c-b699-44df-983a-d58cad14b601-istio-podinfo\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.745851    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-workload-certs\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.746113    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-workload-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.746219    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-istio-envoy\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.746271    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-istio-data\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.746307    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d89616c-b699-44df-983a-d58cad14b601-istio-token\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.746315    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-workload-certs\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.746338    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d89616c-b699-44df-983a-d58cad14b601-istiod-ca-cert\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.746397    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-credential-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.746458    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l994m\" (UniqueName: \"kubernetes.io/projected/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-kube-api-access-l994m\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.746483    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-podinfo\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.746491    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-workload-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.746531    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-token\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.746582    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-credential-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.746605    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istiod-ca-cert\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.746649    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-data\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.747095    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d89616c-b699-44df-983a-d58cad14b601-istiod-ca-cert\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.747202    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-credential-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.747439    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istiod-ca-cert\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.747719    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-istio-data\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.749128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.747756    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-credential-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.750224 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.749217    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-istio-envoy\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.750224 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.749877    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-envoy\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.750621 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.750595    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d89616c-b699-44df-983a-d58cad14b601-istio-podinfo\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.751371 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.751350    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-podinfo\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.805299 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.805245    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-token\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.807036 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.807010    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l994m\" (UniqueName: \"kubernetes.io/projected/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-kube-api-access-l994m\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:55.807407 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.807388    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d89616c-b699-44df-983a-d58cad14b601-istio-token\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.818324 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.817436    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jttf5\" (UniqueName: \"kubernetes.io/projected/3d89616c-b699-44df-983a-d58cad14b601-kube-api-access-jttf5\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:55.833485 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:55.833445    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:56.059382 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:56.059352    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"]
Oct 30 18:58:56.060893 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:58:56.060866    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb82dd7d_e682_42a2_924e_a84f21172fc8.slice/crio-e1ff55590e369e85ea46103efb907a1c23e8b782fff4b8b01aa32ecccbb12466 WatchSource:0}: Error finding container e1ff55590e369e85ea46103efb907a1c23e8b782fff4b8b01aa32ecccbb12466: Status 404 returned error can't find the container with id e1ff55590e369e85ea46103efb907a1c23e8b782fff4b8b01aa32ecccbb12466
Oct 30 18:58:56.073518 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:56.073494    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:56.212893 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:56.212861    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5eaa0bd-lrwlw"
Oct 30 18:58:56.212893 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:56.212877    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5eaa0bd-lrwlw" event={"ID":"88059f5b-33a7-407d-8c56-5656fc90cace","Type":"ContainerDied","Data":"690a225717e21fa83c14be0071fe40994c3f519dbc16c7113cfb72f74f09869c"}
Oct 30 18:58:56.213166 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:56.212932    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="690a225717e21fa83c14be0071fe40994c3f519dbc16c7113cfb72f74f09869c"
Oct 30 18:58:56.222176 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:56.221483    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" event={"ID":"eb82dd7d-e682-42a2-924e-a84f21172fc8","Type":"ContainerStarted","Data":"bc190554447ca2a2b0675d2f7cb195b5e207fc1dc5122bd75aa59008c6ef1ab3"}
Oct 30 18:58:56.222176 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:56.221522    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" event={"ID":"eb82dd7d-e682-42a2-924e-a84f21172fc8","Type":"ContainerStarted","Data":"e1ff55590e369e85ea46103efb907a1c23e8b782fff4b8b01aa32ecccbb12466"}
Oct 30 18:58:56.251624 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:58:56.251590    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d89616c_b699_44df_983a_d58cad14b601.slice/crio-b3ab0d9e08f7374c8c3e8d8261b532d55feeab110f50e6ae125e43a95a9dd865 WatchSource:0}: Error finding container b3ab0d9e08f7374c8c3e8d8261b532d55feeab110f50e6ae125e43a95a9dd865: Status 404 returned error can't find the container with id b3ab0d9e08f7374c8c3e8d8261b532d55feeab110f50e6ae125e43a95a9dd865
Oct 30 18:58:56.253710 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:56.253678    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"]
Oct 30 18:58:56.254359 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:56.254329    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:56.254447 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:56.254392    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:56.254447 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:56.254430    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:56.345246 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:56.345196    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" podStartSLOduration=1.345181449 podStartE2EDuration="1.345181449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:56.341160331 +0000 UTC m=+1337.618210772" watchObservedRunningTime="2026-10-30 18:58:56.345181449 +0000 UTC m=+1337.622231893"
Oct 30 18:58:56.469594 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:56.468933    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"]
Oct 30 18:58:56.496622 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:56.496583    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:56.496770 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:56.496680    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:56.496770 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:56.496725    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 18:58:56.672641 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:56.672600    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:58:57.258195 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:57.258157    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc" event={"ID":"3d89616c-b699-44df-983a-d58cad14b601","Type":"ContainerStarted","Data":"69adc337b05af0ac65343a1ac235e49fc34c7a31d79624240d838473650d589a"}
Oct 30 18:58:57.258399 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:57.258221    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc" event={"ID":"3d89616c-b699-44df-983a-d58cad14b601","Type":"ContainerStarted","Data":"b3ab0d9e08f7374c8c3e8d8261b532d55feeab110f50e6ae125e43a95a9dd865"}
Oct 30 18:58:57.260306 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:57.260278    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw" event={"ID":"ccd81788-05eb-4696-938f-d6dc6d3ab2ce","Type":"ContainerStarted","Data":"a20ef174b15c0d5c8dd4912907e77576e9907973a6773c5a6bbb2f624b806b7b"}
Oct 30 18:58:57.260441 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:57.260321    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw" event={"ID":"ccd81788-05eb-4696-938f-d6dc6d3ab2ce","Type":"ContainerStarted","Data":"db2a7da134ec1e27348f23a484be62990061c4df722ea179040640bb0f8da161"}
Oct 30 18:58:57.630089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:57.629824    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw" podStartSLOduration=2.629798988 podStartE2EDuration="2.629798988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:57.615560827 +0000 UTC m=+1338.892611273" watchObservedRunningTime="2026-10-30 18:58:57.629798988 +0000 UTC m=+1338.906849437"
Oct 30 18:58:57.639089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:57.637777    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc" podStartSLOduration=2.637755093 podStartE2EDuration="2.637755093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:57.482052496 +0000 UTC m=+1338.759102953" watchObservedRunningTime="2026-10-30 18:58:57.637755093 +0000 UTC m=+1338.914805539"
Oct 30 18:58:57.836987 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:57.836949    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:58:57.838084 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:57.838034    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 18:58:57.838276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:57.838253    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc" podUID="3d89616c-b699-44df-983a-d58cad14b601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 18:58:58.068944 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:58.068850    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74304d45-69fcr"]
Oct 30 18:58:58.073225 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:58.073200    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74304d45-69fcr"]
Oct 30 18:58:58.074128 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:58.074103    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:58:58.075810 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:58.075789    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 18:58:58.075903 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:58.075833    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw" podUID="ccd81788-05eb-4696-938f-d6dc6d3ab2ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 18:58:58.456090 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:58.455939    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5486fdf2-2wmdg"]
Oct 30 18:58:58.460939 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:58.460917    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5486fdf2-2wmdg"
Oct 30 18:58:58.487660 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:58.487631    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5486fdf2-2wmdg"]
Oct 30 18:58:58.556413 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:58.554902    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:58:58.556413 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:58.555121    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 18:58:58.556413 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:58.555792    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:58:58.556413 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:58.555850    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:58:58.559006 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:58.558983    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da83659c-89d9-4f6b-8d61-640f7246cb87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5486fdf2-2wmdg\" (UID: \"da83659c-89d9-4f6b-8d61-640f7246cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5486fdf2-2wmdg"
Oct 30 18:58:58.559286 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:58.559269    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8vcb\" (UniqueName: \"kubernetes.io/projected/da83659c-89d9-4f6b-8d61-640f7246cb87-kube-api-access-r8vcb\") pod \"maas-api-revoke-sub-5486fdf2-2wmdg\" (UID: \"da83659c-89d9-4f6b-8d61-640f7246cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5486fdf2-2wmdg"
Oct 30 18:58:58.559401 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:58.559391    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da83659c-89d9-4f6b-8d61-640f7246cb87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5486fdf2-2wmdg\" (UID: \"da83659c-89d9-4f6b-8d61-640f7246cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5486fdf2-2wmdg"
Oct 30 18:58:58.660985 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:58.660943    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da83659c-89d9-4f6b-8d61-640f7246cb87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5486fdf2-2wmdg\" (UID: \"da83659c-89d9-4f6b-8d61-640f7246cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5486fdf2-2wmdg"
Oct 30 18:58:58.661411 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:58.661056    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8vcb\" (UniqueName: \"kubernetes.io/projected/da83659c-89d9-4f6b-8d61-640f7246cb87-kube-api-access-r8vcb\") pod \"maas-api-revoke-sub-5486fdf2-2wmdg\" (UID: \"da83659c-89d9-4f6b-8d61-640f7246cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5486fdf2-2wmdg"
Oct 30 18:58:58.661411 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:58.661130    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da83659c-89d9-4f6b-8d61-640f7246cb87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5486fdf2-2wmdg\" (UID: \"da83659c-89d9-4f6b-8d61-640f7246cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5486fdf2-2wmdg"
Oct 30 18:58:58.662226 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:58.662205    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da83659c-89d9-4f6b-8d61-640f7246cb87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5486fdf2-2wmdg\" (UID: \"da83659c-89d9-4f6b-8d61-640f7246cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5486fdf2-2wmdg"
Oct 30 18:58:58.663735 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:58.663567    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da83659c-89d9-4f6b-8d61-640f7246cb87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5486fdf2-2wmdg\" (UID: \"da83659c-89d9-4f6b-8d61-640f7246cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5486fdf2-2wmdg"
Oct 30 18:58:58.747139 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:58.746835    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8vcb\" (UniqueName: \"kubernetes.io/projected/da83659c-89d9-4f6b-8d61-640f7246cb87-kube-api-access-r8vcb\") pod \"maas-api-revoke-sub-5486fdf2-2wmdg\" (UID: \"da83659c-89d9-4f6b-8d61-640f7246cb87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5486fdf2-2wmdg"
Oct 30 18:58:58.777603 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:58.777576    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5486fdf2-2wmdg"
Oct 30 18:58:58.854088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:58.853360    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 18:58:58.854088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:58.853421    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc" podUID="3d89616c-b699-44df-983a-d58cad14b601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 18:58:58.880604 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:58.880562    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f645db0-9c78-4e62-b10e-1724821f9315" path="/var/lib/kubelet/pods/0f645db0-9c78-4e62-b10e-1724821f9315/volumes"
Oct 30 18:58:59.074829 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:59.074749    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 18:58:59.074829 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:59.074799    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw" podUID="ccd81788-05eb-4696-938f-d6dc6d3ab2ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 18:58:59.237089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:59.236723    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5486fdf2-2wmdg"]
Oct 30 18:58:59.257131 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:58:59.257054    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda83659c_89d9_4f6b_8d61_640f7246cb87.slice/crio-3da1bec9892974b0d016bfaae8200e3c7e7727ebf347f629e91ad73a73a96c82 WatchSource:0}: Error finding container 3da1bec9892974b0d016bfaae8200e3c7e7727ebf347f629e91ad73a73a96c82: Status 404 returned error can't find the container with id 3da1bec9892974b0d016bfaae8200e3c7e7727ebf347f629e91ad73a73a96c82
Oct 30 18:58:59.285680 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:59.285645    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5486fdf2-2wmdg" event={"ID":"da83659c-89d9-4f6b-8d61-640f7246cb87","Type":"ContainerStarted","Data":"3da1bec9892974b0d016bfaae8200e3c7e7727ebf347f629e91ad73a73a96c82"}
Oct 30 18:58:59.852179 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:59.848685    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 18:58:59.852179 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:58:59.848753    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc" podUID="3d89616c-b699-44df-983a-d58cad14b601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 18:59:00.074177 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:00.074136    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 18:59:00.074344 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:00.074197    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw" podUID="ccd81788-05eb-4696-938f-d6dc6d3ab2ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 18:59:00.370089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:00.364833    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5486fdf2-2wmdg" event={"ID":"da83659c-89d9-4f6b-8d61-640f7246cb87","Type":"ContainerStarted","Data":"187eab82d11a8011329b5c35f49eac97fea68789091622aada4b101bb17ea909"}
Oct 30 18:59:00.501230 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:00.492170    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5486fdf2-2wmdg" podStartSLOduration=2.492152394 podStartE2EDuration="2.492152394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:00.481115585 +0000 UTC m=+1341.758166030" watchObservedRunningTime="2026-10-30 18:59:00.492152394 +0000 UTC m=+1341.769202841"
Oct 30 18:59:00.845141 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:00.836180    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 18:59:00.845141 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:00.836251    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc" podUID="3d89616c-b699-44df-983a-d58cad14b601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 18:59:01.076088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:01.075576    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 18:59:01.076088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:01.075670    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw" podUID="ccd81788-05eb-4696-938f-d6dc6d3ab2ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 18:59:01.415393 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:01.415311    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:01.417813 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:01.417212    4924 generic.go:356] "Generic (PLEG): container finished" podID="da83659c-89d9-4f6b-8d61-640f7246cb87" containerID="187eab82d11a8011329b5c35f49eac97fea68789091622aada4b101bb17ea909" exitCode=0
Oct 30 18:59:01.417813 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:01.417354    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5486fdf2-2wmdg" event={"ID":"da83659c-89d9-4f6b-8d61-640f7246cb87","Type":"ContainerDied","Data":"187eab82d11a8011329b5c35f49eac97fea68789091622aada4b101bb17ea909"}
Oct 30 18:59:01.855769 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:01.855731    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 18:59:01.856044 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:01.856017    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc" podUID="3d89616c-b699-44df-983a-d58cad14b601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 18:59:01.998281 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:01.997174    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2" podUID="fdf4711c-418a-436b-848a-d46974566573" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:02.074541 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:02.074499    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 18:59:02.074797 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:02.074764    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw" podUID="ccd81788-05eb-4696-938f-d6dc6d3ab2ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 18:59:02.083211 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:02.083142    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj" podUID="51f3e639-9cd4-4aaa-a568-f19b6d2704fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:02.111656 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:02.111547    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m" podUID="1a90412d-c047-422e-b3c7-9546d286c303" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 18:59:02.271274 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:02.271209    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5" podUID="925a4de9-b451-4263-9a1c-eebbbae4a08c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:02.314271 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:02.314211    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86" podUID="60a58597-fdc4-4722-8d2a-29e475f84a76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:02.445271 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:02.444237    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld" podUID="f04af09f-b8d2-4b47-b0b3-88eb12adff93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:02.852711 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:02.852666    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 18:59:02.852915 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:02.852734    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc" podUID="3d89616c-b699-44df-983a-d58cad14b601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 18:59:02.866195 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:02.866162    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5486fdf2-2wmdg"
Oct 30 18:59:02.973488 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:02.973457    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da83659c-89d9-4f6b-8d61-640f7246cb87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da83659c-89d9-4f6b-8d61-640f7246cb87-maas-api-openshift-ca\") pod \"da83659c-89d9-4f6b-8d61-640f7246cb87\" (UID: \"da83659c-89d9-4f6b-8d61-640f7246cb87\") "
Oct 30 18:59:02.973656 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:02.973599    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da83659c-89d9-4f6b-8d61-640f7246cb87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da83659c-89d9-4f6b-8d61-640f7246cb87-maas-api-xks-ca\") pod \"da83659c-89d9-4f6b-8d61-640f7246cb87\" (UID: \"da83659c-89d9-4f6b-8d61-640f7246cb87\") "
Oct 30 18:59:02.973656 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:02.973629    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da83659c-89d9-4f6b-8d61-640f7246cb87-kube-api-access-r8vcb\" (UniqueName: \"kubernetes.io/projected/da83659c-89d9-4f6b-8d61-640f7246cb87-kube-api-access-r8vcb\") pod \"da83659c-89d9-4f6b-8d61-640f7246cb87\" (UID: \"da83659c-89d9-4f6b-8d61-640f7246cb87\") "
Oct 30 18:59:02.973831 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:02.973806    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da83659c-89d9-4f6b-8d61-640f7246cb87-maas-api-openshift-ca" pod "da83659c-89d9-4f6b-8d61-640f7246cb87" (UID: "da83659c-89d9-4f6b-8d61-640f7246cb87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:02.974055 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:02.974037    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da83659c-89d9-4f6b-8d61-640f7246cb87-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:02.975663 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:02.975631    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da83659c-89d9-4f6b-8d61-640f7246cb87-maas-api-xks-ca" pod "da83659c-89d9-4f6b-8d61-640f7246cb87" (UID: "da83659c-89d9-4f6b-8d61-640f7246cb87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:02.975844 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:02.975826    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da83659c-89d9-4f6b-8d61-640f7246cb87-kube-api-access-r8vcb" pod "da83659c-89d9-4f6b-8d61-640f7246cb87" (UID: "da83659c-89d9-4f6b-8d61-640f7246cb87"). InnerVolumeSpecName "kube-api-access-r8vcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:03.073997 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:03.073906    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Oct 30 18:59:03.073997 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:03.073971    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw" podUID="ccd81788-05eb-4696-938f-d6dc6d3ab2ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Oct 30 18:59:03.074979 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:03.074952    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da83659c-89d9-4f6b-8d61-640f7246cb87-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:03.075110 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:03.075007    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8vcb\" (UniqueName: \"kubernetes.io/projected/da83659c-89d9-4f6b-8d61-640f7246cb87-kube-api-access-r8vcb\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:03.456865 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:03.456834    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5486fdf2-2wmdg"
Oct 30 18:59:03.456865 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:03.456857    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5486fdf2-2wmdg" event={"ID":"da83659c-89d9-4f6b-8d61-640f7246cb87","Type":"ContainerDied","Data":"3da1bec9892974b0d016bfaae8200e3c7e7727ebf347f629e91ad73a73a96c82"}
Oct 30 18:59:03.457474 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:03.456932    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3da1bec9892974b0d016bfaae8200e3c7e7727ebf347f629e91ad73a73a96c82"
Oct 30 18:59:03.834121 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:03.834021    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 18:59:03.834296 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:03.834185    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc" podUID="3d89616c-b699-44df-983a-d58cad14b601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 18:59:04.088995 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.088901    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db6f995c-4s4kh"]
Oct 30 18:59:04.090199 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.090172    4924 cpu_manager.go:414] "containerMap: removing container" podUID="da83659c-89d9-4f6b-8d61-640f7246cb87" containerName="revoke-keys"
Oct 30 18:59:04.090419 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.090400    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da83659c-89d9-4f6b-8d61-640f7246cb87" containerName="revoke-keys"
Oct 30 18:59:04.090751 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.090732    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="da83659c-89d9-4f6b-8d61-640f7246cb87" containerName="revoke-keys"
Oct 30 18:59:04.095276 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.095257    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db6f995c-4s4kh"
Oct 30 18:59:04.168651 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.160376    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db6f995c-4s4kh"]
Oct 30 18:59:04.198646 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.190915    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b61eb9a8-e0da-4a58-97ae-3c96292a14ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db6f995c-4s4kh\" (UID: \"b61eb9a8-e0da-4a58-97ae-3c96292a14ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db6f995c-4s4kh"
Oct 30 18:59:04.198646 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.191177    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-577jr\" (UniqueName: \"kubernetes.io/projected/b61eb9a8-e0da-4a58-97ae-3c96292a14ec-kube-api-access-577jr\") pod \"maas-api-revoke-sub-db6f995c-4s4kh\" (UID: \"b61eb9a8-e0da-4a58-97ae-3c96292a14ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db6f995c-4s4kh"
Oct 30 18:59:04.198646 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.191229    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b61eb9a8-e0da-4a58-97ae-3c96292a14ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db6f995c-4s4kh\" (UID: \"b61eb9a8-e0da-4a58-97ae-3c96292a14ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db6f995c-4s4kh"
Oct 30 18:59:04.198646 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.196331    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:59:04.198646 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.196492    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 18:59:04.198646 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.196637    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:59:04.198646 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.196739    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:59:04.292740 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.292701    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-577jr\" (UniqueName: \"kubernetes.io/projected/b61eb9a8-e0da-4a58-97ae-3c96292a14ec-kube-api-access-577jr\") pod \"maas-api-revoke-sub-db6f995c-4s4kh\" (UID: \"b61eb9a8-e0da-4a58-97ae-3c96292a14ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db6f995c-4s4kh"
Oct 30 18:59:04.292962 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.292765    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b61eb9a8-e0da-4a58-97ae-3c96292a14ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db6f995c-4s4kh\" (UID: \"b61eb9a8-e0da-4a58-97ae-3c96292a14ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db6f995c-4s4kh"
Oct 30 18:59:04.292962 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.292880    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b61eb9a8-e0da-4a58-97ae-3c96292a14ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db6f995c-4s4kh\" (UID: \"b61eb9a8-e0da-4a58-97ae-3c96292a14ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db6f995c-4s4kh"
Oct 30 18:59:04.293849 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.293822    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b61eb9a8-e0da-4a58-97ae-3c96292a14ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db6f995c-4s4kh\" (UID: \"b61eb9a8-e0da-4a58-97ae-3c96292a14ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db6f995c-4s4kh"
Oct 30 18:59:04.295933 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.295908    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b61eb9a8-e0da-4a58-97ae-3c96292a14ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db6f995c-4s4kh\" (UID: \"b61eb9a8-e0da-4a58-97ae-3c96292a14ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db6f995c-4s4kh"
Oct 30 18:59:04.349297 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.349259    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-577jr\" (UniqueName: \"kubernetes.io/projected/b61eb9a8-e0da-4a58-97ae-3c96292a14ec-kube-api-access-577jr\") pod \"maas-api-revoke-sub-db6f995c-4s4kh\" (UID: \"b61eb9a8-e0da-4a58-97ae-3c96292a14ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db6f995c-4s4kh"
Oct 30 18:59:04.422297 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.422248    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db6f995c-4s4kh"
Oct 30 18:59:04.496603 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.496391    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:59:04.497631 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.497609    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:59:04.498908 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.498884    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 18:59:04.822762 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:59:04.822731    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61eb9a8_e0da_4a58_97ae_3c96292a14ec.slice/crio-0aa58341b88f9fad5c3831bd977993f321707542c1bf9831691c6f793974dfc3 WatchSource:0}: Error finding container 0aa58341b88f9fad5c3831bd977993f321707542c1bf9831691c6f793974dfc3: Status 404 returned error can't find the container with id 0aa58341b88f9fad5c3831bd977993f321707542c1bf9831691c6f793974dfc3
Oct 30 18:59:04.823148 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.823128    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db6f995c-4s4kh"]
Oct 30 18:59:04.840002 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.839980    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:59:04.840456 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.840416    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:59:04.841183 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:04.841159    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 18:59:05.482772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:05.482679    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db6f995c-4s4kh" event={"ID":"b61eb9a8-e0da-4a58-97ae-3c96292a14ec","Type":"ContainerStarted","Data":"9a208834f15043265a1273ff093d0e932caa2f881ec11cfdf89225207f4c0959"}
Oct 30 18:59:05.482772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:05.482756    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db6f995c-4s4kh" event={"ID":"b61eb9a8-e0da-4a58-97ae-3c96292a14ec","Type":"ContainerStarted","Data":"0aa58341b88f9fad5c3831bd977993f321707542c1bf9831691c6f793974dfc3"}
Oct 30 18:59:05.600570 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:05.600507    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db6f995c-4s4kh" podStartSLOduration=1.600488712 podStartE2EDuration="1.600488712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:05.598355443 +0000 UTC m=+1346.875405891" watchObservedRunningTime="2026-10-30 18:59:05.600488712 +0000 UTC m=+1346.877539158"
Oct 30 18:59:05.673023 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:05.672996    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:59:05.685925 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:05.685904    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:59:06.049357 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:06.049318    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcb25626-pr6sm"]
Oct 30 18:59:06.053716 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:06.053660    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcb25626-pr6sm"]
Oct 30 18:59:06.492164 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:06.492133    4924 generic.go:356] "Generic (PLEG): container finished" podID="b61eb9a8-e0da-4a58-97ae-3c96292a14ec" containerID="9a208834f15043265a1273ff093d0e932caa2f881ec11cfdf89225207f4c0959" exitCode=0
Oct 30 18:59:06.492359 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:06.492222    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db6f995c-4s4kh" event={"ID":"b61eb9a8-e0da-4a58-97ae-3c96292a14ec","Type":"ContainerDied","Data":"9a208834f15043265a1273ff093d0e932caa2f881ec11cfdf89225207f4c0959"}
Oct 30 18:59:06.503800 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:06.503779    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:59:06.581721 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:06.581646    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:06.838633 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:06.838551    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8b2909a-6460-47d2-99f2-7fe93b21e81c" path="/var/lib/kubelet/pods/e8b2909a-6460-47d2-99f2-7fe93b21e81c/volumes"
Oct 30 18:59:07.649889 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:07.649865    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db6f995c-4s4kh"
Oct 30 18:59:07.740610 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:07.740578    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b61eb9a8-e0da-4a58-97ae-3c96292a14ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b61eb9a8-e0da-4a58-97ae-3c96292a14ec-maas-api-xks-ca\") pod \"b61eb9a8-e0da-4a58-97ae-3c96292a14ec\" (UID: \"b61eb9a8-e0da-4a58-97ae-3c96292a14ec\") "
Oct 30 18:59:07.740765 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:07.740694    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b61eb9a8-e0da-4a58-97ae-3c96292a14ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b61eb9a8-e0da-4a58-97ae-3c96292a14ec-maas-api-openshift-ca\") pod \"b61eb9a8-e0da-4a58-97ae-3c96292a14ec\" (UID: \"b61eb9a8-e0da-4a58-97ae-3c96292a14ec\") "
Oct 30 18:59:07.740765 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:07.740731    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61eb9a8-e0da-4a58-97ae-3c96292a14ec-kube-api-access-577jr\" (UniqueName: \"kubernetes.io/projected/b61eb9a8-e0da-4a58-97ae-3c96292a14ec-kube-api-access-577jr\") pod \"b61eb9a8-e0da-4a58-97ae-3c96292a14ec\" (UID: \"b61eb9a8-e0da-4a58-97ae-3c96292a14ec\") "
Oct 30 18:59:07.741136 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:07.741103    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61eb9a8-e0da-4a58-97ae-3c96292a14ec-maas-api-openshift-ca" pod "b61eb9a8-e0da-4a58-97ae-3c96292a14ec" (UID: "b61eb9a8-e0da-4a58-97ae-3c96292a14ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:07.742770 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:07.742742    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61eb9a8-e0da-4a58-97ae-3c96292a14ec-maas-api-xks-ca" pod "b61eb9a8-e0da-4a58-97ae-3c96292a14ec" (UID: "b61eb9a8-e0da-4a58-97ae-3c96292a14ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:07.742877 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:07.742825    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61eb9a8-e0da-4a58-97ae-3c96292a14ec-kube-api-access-577jr" pod "b61eb9a8-e0da-4a58-97ae-3c96292a14ec" (UID: "b61eb9a8-e0da-4a58-97ae-3c96292a14ec"). InnerVolumeSpecName "kube-api-access-577jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:07.842261 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:07.842224    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b61eb9a8-e0da-4a58-97ae-3c96292a14ec-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:07.842261 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:07.842262    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-577jr\" (UniqueName: \"kubernetes.io/projected/b61eb9a8-e0da-4a58-97ae-3c96292a14ec-kube-api-access-577jr\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:07.842799 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:07.842279    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b61eb9a8-e0da-4a58-97ae-3c96292a14ec-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:08.511662 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:08.511631    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db6f995c-4s4kh" event={"ID":"b61eb9a8-e0da-4a58-97ae-3c96292a14ec","Type":"ContainerDied","Data":"0aa58341b88f9fad5c3831bd977993f321707542c1bf9831691c6f793974dfc3"}
Oct 30 18:59:08.511836 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:08.511680    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0aa58341b88f9fad5c3831bd977993f321707542c1bf9831691c6f793974dfc3"
Oct 30 18:59:08.511836 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:08.511688    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db6f995c-4s4kh"
Oct 30 18:59:09.079836 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:09.079802    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8779c64-8klw8"]
Oct 30 18:59:09.080672 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:09.080649    4924 cpu_manager.go:414] "containerMap: removing container" podUID="b61eb9a8-e0da-4a58-97ae-3c96292a14ec" containerName="revoke-keys"
Oct 30 18:59:09.080781 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:09.080672    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61eb9a8-e0da-4a58-97ae-3c96292a14ec" containerName="revoke-keys"
Oct 30 18:59:09.080852 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:09.080837    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61eb9a8-e0da-4a58-97ae-3c96292a14ec" containerName="revoke-keys"
Oct 30 18:59:09.084599 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:09.084580    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8779c64-8klw8"
Oct 30 18:59:09.134407 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:09.134384    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:59:09.134697 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:09.134683    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 18:59:09.158922 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:09.158897    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a38942b6-7d30-4535-8be2-94542d7c8776-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8779c64-8klw8\" (UID: \"a38942b6-7d30-4535-8be2-94542d7c8776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8779c64-8klw8"
Oct 30 18:59:09.159113 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:09.159091    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmjln\" (UniqueName: \"kubernetes.io/projected/a38942b6-7d30-4535-8be2-94542d7c8776-kube-api-access-tmjln\") pod \"maas-api-revoke-sub-a8779c64-8klw8\" (UID: \"a38942b6-7d30-4535-8be2-94542d7c8776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8779c64-8klw8"
Oct 30 18:59:09.159209 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:09.159170    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a38942b6-7d30-4535-8be2-94542d7c8776-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8779c64-8klw8\" (UID: \"a38942b6-7d30-4535-8be2-94542d7c8776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8779c64-8klw8"
Oct 30 18:59:09.181299 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:09.181274    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:59:09.181416 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:09.181330    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:59:09.186223 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:09.186202    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8779c64-8klw8"]
Oct 30 18:59:09.260957 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:09.260922    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a38942b6-7d30-4535-8be2-94542d7c8776-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8779c64-8klw8\" (UID: \"a38942b6-7d30-4535-8be2-94542d7c8776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8779c64-8klw8"
Oct 30 18:59:09.261151 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:09.261025    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmjln\" (UniqueName: \"kubernetes.io/projected/a38942b6-7d30-4535-8be2-94542d7c8776-kube-api-access-tmjln\") pod \"maas-api-revoke-sub-a8779c64-8klw8\" (UID: \"a38942b6-7d30-4535-8be2-94542d7c8776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8779c64-8klw8"
Oct 30 18:59:09.261151 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:09.261082    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a38942b6-7d30-4535-8be2-94542d7c8776-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8779c64-8klw8\" (UID: \"a38942b6-7d30-4535-8be2-94542d7c8776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8779c64-8klw8"
Oct 30 18:59:09.261652 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:09.261630    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a38942b6-7d30-4535-8be2-94542d7c8776-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8779c64-8klw8\" (UID: \"a38942b6-7d30-4535-8be2-94542d7c8776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8779c64-8klw8"
Oct 30 18:59:09.263407 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:09.263389    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a38942b6-7d30-4535-8be2-94542d7c8776-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8779c64-8klw8\" (UID: \"a38942b6-7d30-4535-8be2-94542d7c8776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8779c64-8klw8"
Oct 30 18:59:09.315881 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:09.315856    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmjln\" (UniqueName: \"kubernetes.io/projected/a38942b6-7d30-4535-8be2-94542d7c8776-kube-api-access-tmjln\") pod \"maas-api-revoke-sub-a8779c64-8klw8\" (UID: \"a38942b6-7d30-4535-8be2-94542d7c8776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8779c64-8klw8"
Oct 30 18:59:09.401575 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:09.401549    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8779c64-8klw8"
Oct 30 18:59:09.777556 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:59:09.777525    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda38942b6_7d30_4535_8be2_94542d7c8776.slice/crio-afd8448724f1b54089881b225f419bd573b580082836e5cb921c5340b4f30c72 WatchSource:0}: Error finding container afd8448724f1b54089881b225f419bd573b580082836e5cb921c5340b4f30c72: Status 404 returned error can't find the container with id afd8448724f1b54089881b225f419bd573b580082836e5cb921c5340b4f30c72
Oct 30 18:59:09.778043 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:09.778022    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8779c64-8klw8"]
Oct 30 18:59:10.536291 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:10.536202    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8779c64-8klw8" event={"ID":"a38942b6-7d30-4535-8be2-94542d7c8776","Type":"ContainerStarted","Data":"632b1c35d54836a1ec02f5c024a4f6587eb23702d04a4a56c2ce2b792e60853f"}
Oct 30 18:59:10.536291 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:10.536272    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8779c64-8klw8" event={"ID":"a38942b6-7d30-4535-8be2-94542d7c8776","Type":"ContainerStarted","Data":"afd8448724f1b54089881b225f419bd573b580082836e5cb921c5340b4f30c72"}
Oct 30 18:59:10.646926 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:10.646858    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8779c64-8klw8" podStartSLOduration=1.6468329910000001 podStartE2EDuration="1.646832991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:10.644846595 +0000 UTC m=+1351.921897056" watchObservedRunningTime="2026-10-30 18:59:10.646832991 +0000 UTC m=+1351.923883437"
Oct 30 18:59:11.532474 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:11.532439    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:11.549192 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:11.549155    4924 generic.go:356] "Generic (PLEG): container finished" podID="a38942b6-7d30-4535-8be2-94542d7c8776" containerID="632b1c35d54836a1ec02f5c024a4f6587eb23702d04a4a56c2ce2b792e60853f" exitCode=0
Oct 30 18:59:11.549696 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:11.549289    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8779c64-8klw8" event={"ID":"a38942b6-7d30-4535-8be2-94542d7c8776","Type":"ContainerDied","Data":"632b1c35d54836a1ec02f5c024a4f6587eb23702d04a4a56c2ce2b792e60853f"}
Oct 30 18:59:12.395945 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:12.395915    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:12.708946 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:12.708922    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8779c64-8klw8"
Oct 30 18:59:12.806173 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:12.806146    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a38942b6-7d30-4535-8be2-94542d7c8776-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a38942b6-7d30-4535-8be2-94542d7c8776-maas-api-xks-ca\") pod \"a38942b6-7d30-4535-8be2-94542d7c8776\" (UID: \"a38942b6-7d30-4535-8be2-94542d7c8776\") "
Oct 30 18:59:12.806302 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:12.806193    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a38942b6-7d30-4535-8be2-94542d7c8776-kube-api-access-tmjln\" (UniqueName: \"kubernetes.io/projected/a38942b6-7d30-4535-8be2-94542d7c8776-kube-api-access-tmjln\") pod \"a38942b6-7d30-4535-8be2-94542d7c8776\" (UID: \"a38942b6-7d30-4535-8be2-94542d7c8776\") "
Oct 30 18:59:12.806302 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:12.806218    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a38942b6-7d30-4535-8be2-94542d7c8776-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a38942b6-7d30-4535-8be2-94542d7c8776-maas-api-openshift-ca\") pod \"a38942b6-7d30-4535-8be2-94542d7c8776\" (UID: \"a38942b6-7d30-4535-8be2-94542d7c8776\") "
Oct 30 18:59:12.806517 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:12.806492    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a38942b6-7d30-4535-8be2-94542d7c8776-maas-api-openshift-ca" pod "a38942b6-7d30-4535-8be2-94542d7c8776" (UID: "a38942b6-7d30-4535-8be2-94542d7c8776"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:12.806772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:12.806751    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a38942b6-7d30-4535-8be2-94542d7c8776-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:12.808266 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:12.808242    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a38942b6-7d30-4535-8be2-94542d7c8776-maas-api-xks-ca" pod "a38942b6-7d30-4535-8be2-94542d7c8776" (UID: "a38942b6-7d30-4535-8be2-94542d7c8776"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:12.808406 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:12.808391    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a38942b6-7d30-4535-8be2-94542d7c8776-kube-api-access-tmjln" pod "a38942b6-7d30-4535-8be2-94542d7c8776" (UID: "a38942b6-7d30-4535-8be2-94542d7c8776"). InnerVolumeSpecName "kube-api-access-tmjln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:12.908126 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:12.908094    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a38942b6-7d30-4535-8be2-94542d7c8776-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:12.908126 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:12.908118    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmjln\" (UniqueName: \"kubernetes.io/projected/a38942b6-7d30-4535-8be2-94542d7c8776-kube-api-access-tmjln\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:13.565545 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:13.565512    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8779c64-8klw8"
Oct 30 18:59:13.565713 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:13.565546    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8779c64-8klw8" event={"ID":"a38942b6-7d30-4535-8be2-94542d7c8776","Type":"ContainerDied","Data":"afd8448724f1b54089881b225f419bd573b580082836e5cb921c5340b4f30c72"}
Oct 30 18:59:13.565713 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:13.565594    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afd8448724f1b54089881b225f419bd573b580082836e5cb921c5340b4f30c72"
Oct 30 18:59:15.711555 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:59:15.711520    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:59:16.500388 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:16.500318    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:17.089814 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:17.089780    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37c03f3a-csvn4"]
Oct 30 18:59:17.135273 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:17.135242    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37c03f3a-csvn4"]
Oct 30 18:59:18.835400 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:18.835365    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f54c3946-3c6c-4aab-876f-f0f8c2817604" path="/var/lib/kubelet/pods/f54c3946-3c6c-4aab-876f-f0f8c2817604/volumes"
Oct 30 18:59:20.496271 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:20.496204    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" podUID="eb82dd7d-e682-42a2-924e-a84f21172fc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:20.521191 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:20.521150    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7" podUID="97a3b018-9083-4b7c-af88-6edddc585d67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 18:59:20.540268 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:20.540210    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt" podUID="955e98b9-864b-4636-9511-8b69f981eaa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:20.701171 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:20.700152    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l" podUID="bdc8df4b-8c94-40b8-b4be-80d1432d87de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:20.787147 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:20.787029    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd" podUID="a3d792df-177d-4f7a-bba8-f6351175462e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:20.846901 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:20.845345    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd" podUID="03ac954a-d07b-4523-8d5a-a13dffd0bd2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 18:59:20.914250 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:20.914200    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg" podUID="9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:21.008097 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:21.007119    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2" podUID="fdf4711c-418a-436b-848a-d46974566573" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:21.083428 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:21.082154    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj" podUID="51f3e639-9cd4-4aaa-a568-f19b6d2704fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:21.110297 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:21.109451    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m" podUID="1a90412d-c047-422e-b3c7-9546d286c303" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:21.279356 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:21.278149    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5" podUID="925a4de9-b451-4263-9a1c-eebbbae4a08c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:21.314374 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:21.313565    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86" podUID="60a58597-fdc4-4722-8d2a-29e475f84a76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:21.428597 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:21.428551    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld" podUID="f04af09f-b8d2-4b47-b0b3-88eb12adff93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:21.499290 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:21.499248    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" podUID="eb82dd7d-e682-42a2-924e-a84f21172fc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 30 18:59:21.500533 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:21.500508    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:59:21.524195 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:21.524150    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7" podUID="97a3b018-9083-4b7c-af88-6edddc585d67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:21.524850 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:21.524825    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:59:21.545090 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:21.544140    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt" podUID="955e98b9-864b-4636-9511-8b69f981eaa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:21.545090 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:21.544550    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:59:21.869203 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:21.856212    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd" podUID="03ac954a-d07b-4523-8d5a-a13dffd0bd2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:21.869203 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:21.859055    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l" podUID="bdc8df4b-8c94-40b8-b4be-80d1432d87de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:21.897155 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:21.870097    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:59:21.897155 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:21.870155    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:59:21.965531 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:21.945715    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd" podUID="a3d792df-177d-4f7a-bba8-f6351175462e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:21.965531 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:21.946228    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:59:22.123177 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:22.014002    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2" podUID="fdf4711c-418a-436b-848a-d46974566573" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:22.123177 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:22.016359    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:59:22.123177 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:22.017135    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg" podUID="9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:22.123177 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:22.035407    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:59:22.174088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:22.154237    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m" podUID="1a90412d-c047-422e-b3c7-9546d286c303" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 18:59:22.197269 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:22.175658    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj" podUID="51f3e639-9cd4-4aaa-a568-f19b6d2704fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:22.197269 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:22.178289    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:59:22.197269 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:22.178347    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:59:22.361376 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:22.361328    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86" podUID="60a58597-fdc4-4722-8d2a-29e475f84a76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:22.378147 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:22.363680    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5" podUID="925a4de9-b451-4263-9a1c-eebbbae4a08c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:22.386351 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:22.386322    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:59:22.386631 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:22.386616    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:59:22.455731 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:22.455688    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld" podUID="f04af09f-b8d2-4b47-b0b3-88eb12adff93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:22.457347 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:22.456967    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:59:22.543819 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:22.543364    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7" podUID="97a3b018-9083-4b7c-af88-6edddc585d67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 18:59:22.579360 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:22.555289    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt" podUID="955e98b9-864b-4636-9511-8b69f981eaa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:22.640477 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:22.640375    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" podUID="eb82dd7d-e682-42a2-924e-a84f21172fc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:22.863263 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:22.863215    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l" podUID="bdc8df4b-8c94-40b8-b4be-80d1432d87de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:22.877150 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:22.863518    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd" podUID="03ac954a-d07b-4523-8d5a-a13dffd0bd2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded"
Oct 30 18:59:22.878406 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:22.863548    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd" podUID="03ac954a-d07b-4523-8d5a-a13dffd0bd2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 18:59:22.954144 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:22.954040    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd" podUID="a3d792df-177d-4f7a-bba8-f6351175462e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:23.018184 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:23.018135    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg" podUID="9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:23.058865 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:23.049578    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2" podUID="fdf4711c-418a-436b-848a-d46974566573" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:23.141084 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:23.066693    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:59:23.141084 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:23.066205    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld" podUID="f04af09f-b8d2-4b47-b0b3-88eb12adff93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:23.143171 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:23.143143    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:59:23.168372 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:23.168346    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:59:23.168740 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:23.168723    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:59:23.189123 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:23.170565    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:59:23.190265 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:23.190223    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m" podUID="1a90412d-c047-422e-b3c7-9546d286c303" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:23.284506 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:23.284350    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:59:23.311631 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:23.310324    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:59:23.311631 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:23.311577    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:59:23.316767 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:23.312382    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:59:23.321772 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:23.321647    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:59:23.335101 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:23.334102    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:59:23.335101 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:23.334381    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:59:23.345426 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:23.345403    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:59:23.757722 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:23.757687    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:28.610241 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:28.610208    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dbc97cd-7sg7f"]
Oct 30 18:59:28.610970 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:28.610951    4924 cpu_manager.go:414] "containerMap: removing container" podUID="a38942b6-7d30-4535-8be2-94542d7c8776" containerName="revoke-keys"
Oct 30 18:59:28.611020 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:28.610971    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a38942b6-7d30-4535-8be2-94542d7c8776" containerName="revoke-keys"
Oct 30 18:59:28.611119 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:28.611109    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="a38942b6-7d30-4535-8be2-94542d7c8776" containerName="revoke-keys"
Oct 30 18:59:28.614633 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:28.614619    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbc97cd-7sg7f"
Oct 30 18:59:28.670565 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:28.670535    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dbc97cd-7sg7f"]
Oct 30 18:59:28.674738 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:28.674709    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/470295aa-9133-4e42-bd40-55d6cb373b03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dbc97cd-7sg7f\" (UID: \"470295aa-9133-4e42-bd40-55d6cb373b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbc97cd-7sg7f"
Oct 30 18:59:28.674738 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:28.674740    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77lpl\" (UniqueName: \"kubernetes.io/projected/470295aa-9133-4e42-bd40-55d6cb373b03-kube-api-access-77lpl\") pod \"maas-api-revoke-sub-5dbc97cd-7sg7f\" (UID: \"470295aa-9133-4e42-bd40-55d6cb373b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbc97cd-7sg7f"
Oct 30 18:59:28.674946 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:28.674768    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/470295aa-9133-4e42-bd40-55d6cb373b03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dbc97cd-7sg7f\" (UID: \"470295aa-9133-4e42-bd40-55d6cb373b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbc97cd-7sg7f"
Oct 30 18:59:28.711606 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:28.711586    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:59:28.711812 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:28.711798    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:59:28.711812 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:28.711805    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:59:28.711910 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:28.711845    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 18:59:28.775824 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:28.775800    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/470295aa-9133-4e42-bd40-55d6cb373b03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dbc97cd-7sg7f\" (UID: \"470295aa-9133-4e42-bd40-55d6cb373b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbc97cd-7sg7f"
Oct 30 18:59:28.775968 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:28.775829    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77lpl\" (UniqueName: \"kubernetes.io/projected/470295aa-9133-4e42-bd40-55d6cb373b03-kube-api-access-77lpl\") pod \"maas-api-revoke-sub-5dbc97cd-7sg7f\" (UID: \"470295aa-9133-4e42-bd40-55d6cb373b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbc97cd-7sg7f"
Oct 30 18:59:28.775968 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:28.775856    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/470295aa-9133-4e42-bd40-55d6cb373b03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dbc97cd-7sg7f\" (UID: \"470295aa-9133-4e42-bd40-55d6cb373b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbc97cd-7sg7f"
Oct 30 18:59:28.776443 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:28.776425    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/470295aa-9133-4e42-bd40-55d6cb373b03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dbc97cd-7sg7f\" (UID: \"470295aa-9133-4e42-bd40-55d6cb373b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbc97cd-7sg7f"
Oct 30 18:59:28.778216 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:28.778194    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/470295aa-9133-4e42-bd40-55d6cb373b03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dbc97cd-7sg7f\" (UID: \"470295aa-9133-4e42-bd40-55d6cb373b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbc97cd-7sg7f"
Oct 30 18:59:28.830239 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:28.830209    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77lpl\" (UniqueName: \"kubernetes.io/projected/470295aa-9133-4e42-bd40-55d6cb373b03-kube-api-access-77lpl\") pod \"maas-api-revoke-sub-5dbc97cd-7sg7f\" (UID: \"470295aa-9133-4e42-bd40-55d6cb373b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbc97cd-7sg7f"
Oct 30 18:59:28.879752 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:28.879675    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:28.928352 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:28.928324    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbc97cd-7sg7f"
Oct 30 18:59:29.069274 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:29.069247    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:29.071292 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:29.071263    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72842ed0-ndk5l"]
Oct 30 18:59:29.080110 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:29.080033    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72842ed0-ndk5l"]
Oct 30 18:59:29.337740 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:29.337699    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dbc97cd-7sg7f"]
Oct 30 18:59:29.338507 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:59:29.338480    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod470295aa_9133_4e42_bd40_55d6cb373b03.slice/crio-43d6afce1a3f1e6707417da646e296f6ab852efe8bddf85fba05b59398ffbd95 WatchSource:0}: Error finding container 43d6afce1a3f1e6707417da646e296f6ab852efe8bddf85fba05b59398ffbd95: Status 404 returned error can't find the container with id 43d6afce1a3f1e6707417da646e296f6ab852efe8bddf85fba05b59398ffbd95
Oct 30 18:59:29.448149 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:29.448120    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbc97cd-7sg7f" event={"ID":"470295aa-9133-4e42-bd40-55d6cb373b03","Type":"ContainerStarted","Data":"43d6afce1a3f1e6707417da646e296f6ab852efe8bddf85fba05b59398ffbd95"}
Oct 30 18:59:29.943480 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:29.943386    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c65470d5-wbdgk"]
Oct 30 18:59:29.948392 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:29.948362    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65470d5-wbdgk"
Oct 30 18:59:29.990218 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:29.990186    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8tgq\" (UniqueName: \"kubernetes.io/projected/4b428a6a-2a07-4b92-89cf-4646ed10f619-kube-api-access-b8tgq\") pod \"maas-api-revoke-sub-c65470d5-wbdgk\" (UID: \"4b428a6a-2a07-4b92-89cf-4646ed10f619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65470d5-wbdgk"
Oct 30 18:59:29.990410 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:29.990392    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b428a6a-2a07-4b92-89cf-4646ed10f619-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c65470d5-wbdgk\" (UID: \"4b428a6a-2a07-4b92-89cf-4646ed10f619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65470d5-wbdgk"
Oct 30 18:59:29.990520 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:29.990497    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b428a6a-2a07-4b92-89cf-4646ed10f619-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c65470d5-wbdgk\" (UID: \"4b428a6a-2a07-4b92-89cf-4646ed10f619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65470d5-wbdgk"
Oct 30 18:59:30.003482 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:30.003448    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c65470d5-wbdgk"]
Oct 30 18:59:30.091494 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:30.091459    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b428a6a-2a07-4b92-89cf-4646ed10f619-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c65470d5-wbdgk\" (UID: \"4b428a6a-2a07-4b92-89cf-4646ed10f619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65470d5-wbdgk"
Oct 30 18:59:30.091659 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:30.091548    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b428a6a-2a07-4b92-89cf-4646ed10f619-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c65470d5-wbdgk\" (UID: \"4b428a6a-2a07-4b92-89cf-4646ed10f619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65470d5-wbdgk"
Oct 30 18:59:30.091659 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:30.091594    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8tgq\" (UniqueName: \"kubernetes.io/projected/4b428a6a-2a07-4b92-89cf-4646ed10f619-kube-api-access-b8tgq\") pod \"maas-api-revoke-sub-c65470d5-wbdgk\" (UID: \"4b428a6a-2a07-4b92-89cf-4646ed10f619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65470d5-wbdgk"
Oct 30 18:59:30.092264 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:30.092235    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b428a6a-2a07-4b92-89cf-4646ed10f619-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c65470d5-wbdgk\" (UID: \"4b428a6a-2a07-4b92-89cf-4646ed10f619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65470d5-wbdgk"
Oct 30 18:59:30.093789 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:30.093766    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b428a6a-2a07-4b92-89cf-4646ed10f619-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c65470d5-wbdgk\" (UID: \"4b428a6a-2a07-4b92-89cf-4646ed10f619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65470d5-wbdgk"
Oct 30 18:59:30.146949 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:30.146923    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8tgq\" (UniqueName: \"kubernetes.io/projected/4b428a6a-2a07-4b92-89cf-4646ed10f619-kube-api-access-b8tgq\") pod \"maas-api-revoke-sub-c65470d5-wbdgk\" (UID: \"4b428a6a-2a07-4b92-89cf-4646ed10f619\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65470d5-wbdgk"
Oct 30 18:59:30.266869 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:30.266840    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65470d5-wbdgk"
Oct 30 18:59:30.457143 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:30.457106    4924 generic.go:356] "Generic (PLEG): container finished" podID="470295aa-9133-4e42-bd40-55d6cb373b03" containerID="0c9cbbbcbeb401cac5e2f7b6d6d9aa3457344e6c6cd5f282f5e762fc27e0ae5d" exitCode=0
Oct 30 18:59:30.457328 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:30.457181    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbc97cd-7sg7f" event={"ID":"470295aa-9133-4e42-bd40-55d6cb373b03","Type":"ContainerDied","Data":"0c9cbbbcbeb401cac5e2f7b6d6d9aa3457344e6c6cd5f282f5e762fc27e0ae5d"}
Oct 30 18:59:30.675991 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:59:30.675956    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b428a6a_2a07_4b92_89cf_4646ed10f619.slice/crio-36fdb470165b850117707d8a86a5d40026e1eefc852deeeabb252ce91637b0e2 WatchSource:0}: Error finding container 36fdb470165b850117707d8a86a5d40026e1eefc852deeeabb252ce91637b0e2: Status 404 returned error can't find the container with id 36fdb470165b850117707d8a86a5d40026e1eefc852deeeabb252ce91637b0e2
Oct 30 18:59:30.677662 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:30.677635    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c65470d5-wbdgk"]
Oct 30 18:59:30.836578 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:30.836545    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7d023c1-3461-491d-b3fa-3f3a07f01e7b" path="/var/lib/kubelet/pods/e7d023c1-3461-491d-b3fa-3f3a07f01e7b/volumes"
Oct 30 18:59:31.467376 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:31.467342    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65470d5-wbdgk" event={"ID":"4b428a6a-2a07-4b92-89cf-4646ed10f619","Type":"ContainerStarted","Data":"6a15926b7c2ffd225c8207060f38dca2345439901a444c12c26ba0fd0e05c344"}
Oct 30 18:59:31.467789 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:31.467400    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65470d5-wbdgk" event={"ID":"4b428a6a-2a07-4b92-89cf-4646ed10f619","Type":"ContainerStarted","Data":"36fdb470165b850117707d8a86a5d40026e1eefc852deeeabb252ce91637b0e2"}
Oct 30 18:59:31.595113 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:31.590840    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65470d5-wbdgk" podStartSLOduration=2.590822734 podStartE2EDuration="2.590822734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:31.584406553 +0000 UTC m=+1372.861457014" watchObservedRunningTime="2026-10-30 18:59:31.590822734 +0000 UTC m=+1372.867873182"
Oct 30 18:59:31.646819 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:31.646795    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbc97cd-7sg7f"
Oct 30 18:59:31.715378 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:31.715356    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/470295aa-9133-4e42-bd40-55d6cb373b03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/470295aa-9133-4e42-bd40-55d6cb373b03-maas-api-openshift-ca\") pod \"470295aa-9133-4e42-bd40-55d6cb373b03\" (UID: \"470295aa-9133-4e42-bd40-55d6cb373b03\") "
Oct 30 18:59:31.715494 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:31.715463    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/470295aa-9133-4e42-bd40-55d6cb373b03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/470295aa-9133-4e42-bd40-55d6cb373b03-maas-api-xks-ca\") pod \"470295aa-9133-4e42-bd40-55d6cb373b03\" (UID: \"470295aa-9133-4e42-bd40-55d6cb373b03\") "
Oct 30 18:59:31.715536 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:31.715523    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/470295aa-9133-4e42-bd40-55d6cb373b03-kube-api-access-77lpl\" (UniqueName: \"kubernetes.io/projected/470295aa-9133-4e42-bd40-55d6cb373b03-kube-api-access-77lpl\") pod \"470295aa-9133-4e42-bd40-55d6cb373b03\" (UID: \"470295aa-9133-4e42-bd40-55d6cb373b03\") "
Oct 30 18:59:31.715740 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:31.715708    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/470295aa-9133-4e42-bd40-55d6cb373b03-maas-api-openshift-ca" pod "470295aa-9133-4e42-bd40-55d6cb373b03" (UID: "470295aa-9133-4e42-bd40-55d6cb373b03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:31.716132 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:31.716111    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/470295aa-9133-4e42-bd40-55d6cb373b03-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:31.717539 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:31.717477    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/470295aa-9133-4e42-bd40-55d6cb373b03-maas-api-xks-ca" pod "470295aa-9133-4e42-bd40-55d6cb373b03" (UID: "470295aa-9133-4e42-bd40-55d6cb373b03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:31.717629 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:31.717559    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/470295aa-9133-4e42-bd40-55d6cb373b03-kube-api-access-77lpl" pod "470295aa-9133-4e42-bd40-55d6cb373b03" (UID: "470295aa-9133-4e42-bd40-55d6cb373b03"). InnerVolumeSpecName "kube-api-access-77lpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:31.817630 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:31.817588    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/470295aa-9133-4e42-bd40-55d6cb373b03-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:31.817630 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:31.817623    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77lpl\" (UniqueName: \"kubernetes.io/projected/470295aa-9133-4e42-bd40-55d6cb373b03-kube-api-access-77lpl\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:32.476553 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:32.476523    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbc97cd-7sg7f"
Oct 30 18:59:32.476553 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:32.476548    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dbc97cd-7sg7f" event={"ID":"470295aa-9133-4e42-bd40-55d6cb373b03","Type":"ContainerDied","Data":"43d6afce1a3f1e6707417da646e296f6ab852efe8bddf85fba05b59398ffbd95"}
Oct 30 18:59:32.477055 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:32.476595    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43d6afce1a3f1e6707417da646e296f6ab852efe8bddf85fba05b59398ffbd95"
Oct 30 18:59:32.478249 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:32.478222    4924 generic.go:356] "Generic (PLEG): container finished" podID="4b428a6a-2a07-4b92-89cf-4646ed10f619" containerID="6a15926b7c2ffd225c8207060f38dca2345439901a444c12c26ba0fd0e05c344" exitCode=0
Oct 30 18:59:32.478364 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:32.478252    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65470d5-wbdgk" event={"ID":"4b428a6a-2a07-4b92-89cf-4646ed10f619","Type":"ContainerDied","Data":"6a15926b7c2ffd225c8207060f38dca2345439901a444c12c26ba0fd0e05c344"}
Oct 30 18:59:33.641648 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.641349    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65470d5-wbdgk"
Oct 30 18:59:33.646079 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.646046    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-045c23ec-5s4dt"]
Oct 30 18:59:33.646641 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.646628    4924 cpu_manager.go:414] "containerMap: removing container" podUID="470295aa-9133-4e42-bd40-55d6cb373b03" containerName="revoke-keys"
Oct 30 18:59:33.646702 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.646643    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="470295aa-9133-4e42-bd40-55d6cb373b03" containerName="revoke-keys"
Oct 30 18:59:33.646702 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.646677    4924 cpu_manager.go:414] "containerMap: removing container" podUID="4b428a6a-2a07-4b92-89cf-4646ed10f619" containerName="revoke-keys"
Oct 30 18:59:33.646702 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.646685    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b428a6a-2a07-4b92-89cf-4646ed10f619" containerName="revoke-keys"
Oct 30 18:59:33.646869 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.646794    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="470295aa-9133-4e42-bd40-55d6cb373b03" containerName="revoke-keys"
Oct 30 18:59:33.646869 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.646808    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b428a6a-2a07-4b92-89cf-4646ed10f619" containerName="revoke-keys"
Oct 30 18:59:33.650248 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.650232    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-045c23ec-5s4dt"
Oct 30 18:59:33.741200 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.741115    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b428a6a-2a07-4b92-89cf-4646ed10f619-kube-api-access-b8tgq\" (UniqueName: \"kubernetes.io/projected/4b428a6a-2a07-4b92-89cf-4646ed10f619-kube-api-access-b8tgq\") pod \"4b428a6a-2a07-4b92-89cf-4646ed10f619\" (UID: \"4b428a6a-2a07-4b92-89cf-4646ed10f619\") "
Oct 30 18:59:33.741354 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.741280    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b428a6a-2a07-4b92-89cf-4646ed10f619-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b428a6a-2a07-4b92-89cf-4646ed10f619-maas-api-xks-ca\") pod \"4b428a6a-2a07-4b92-89cf-4646ed10f619\" (UID: \"4b428a6a-2a07-4b92-89cf-4646ed10f619\") "
Oct 30 18:59:33.741474 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.741459    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b428a6a-2a07-4b92-89cf-4646ed10f619-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b428a6a-2a07-4b92-89cf-4646ed10f619-maas-api-openshift-ca\") pod \"4b428a6a-2a07-4b92-89cf-4646ed10f619\" (UID: \"4b428a6a-2a07-4b92-89cf-4646ed10f619\") "
Oct 30 18:59:33.741826 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.741794    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b428a6a-2a07-4b92-89cf-4646ed10f619-maas-api-openshift-ca" pod "4b428a6a-2a07-4b92-89cf-4646ed10f619" (UID: "4b428a6a-2a07-4b92-89cf-4646ed10f619"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:33.741961 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.741925    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4afd685c-067a-40ce-becd-80a838afb556-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-045c23ec-5s4dt\" (UID: \"4afd685c-067a-40ce-becd-80a838afb556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045c23ec-5s4dt"
Oct 30 18:59:33.742058 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.742026    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkgdm\" (UniqueName: \"kubernetes.io/projected/4afd685c-067a-40ce-becd-80a838afb556-kube-api-access-kkgdm\") pod \"maas-api-revoke-sub-045c23ec-5s4dt\" (UID: \"4afd685c-067a-40ce-becd-80a838afb556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045c23ec-5s4dt"
Oct 30 18:59:33.742250 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.742229    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4afd685c-067a-40ce-becd-80a838afb556-maas-api-xks-ca\") pod \"maas-api-revoke-sub-045c23ec-5s4dt\" (UID: \"4afd685c-067a-40ce-becd-80a838afb556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045c23ec-5s4dt"
Oct 30 18:59:33.742324 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.742309    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-045c23ec-5s4dt"]
Oct 30 18:59:33.742391 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.742374    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b428a6a-2a07-4b92-89cf-4646ed10f619-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:33.743390 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.743368    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b428a6a-2a07-4b92-89cf-4646ed10f619-kube-api-access-b8tgq" pod "4b428a6a-2a07-4b92-89cf-4646ed10f619" (UID: "4b428a6a-2a07-4b92-89cf-4646ed10f619"). InnerVolumeSpecName "kube-api-access-b8tgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:33.743468 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.743429    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b428a6a-2a07-4b92-89cf-4646ed10f619-maas-api-xks-ca" pod "4b428a6a-2a07-4b92-89cf-4646ed10f619" (UID: "4b428a6a-2a07-4b92-89cf-4646ed10f619"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:33.843446 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.843409    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4afd685c-067a-40ce-becd-80a838afb556-maas-api-xks-ca\") pod \"maas-api-revoke-sub-045c23ec-5s4dt\" (UID: \"4afd685c-067a-40ce-becd-80a838afb556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045c23ec-5s4dt"
Oct 30 18:59:33.843619 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.843600    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4afd685c-067a-40ce-becd-80a838afb556-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-045c23ec-5s4dt\" (UID: \"4afd685c-067a-40ce-becd-80a838afb556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045c23ec-5s4dt"
Oct 30 18:59:33.843678 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.843643    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkgdm\" (UniqueName: \"kubernetes.io/projected/4afd685c-067a-40ce-becd-80a838afb556-kube-api-access-kkgdm\") pod \"maas-api-revoke-sub-045c23ec-5s4dt\" (UID: \"4afd685c-067a-40ce-becd-80a838afb556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045c23ec-5s4dt"
Oct 30 18:59:33.843751 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.843735    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8tgq\" (UniqueName: \"kubernetes.io/projected/4b428a6a-2a07-4b92-89cf-4646ed10f619-kube-api-access-b8tgq\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:33.843809 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.843757    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b428a6a-2a07-4b92-89cf-4646ed10f619-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:33.844348 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.844320    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4afd685c-067a-40ce-becd-80a838afb556-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-045c23ec-5s4dt\" (UID: \"4afd685c-067a-40ce-becd-80a838afb556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045c23ec-5s4dt"
Oct 30 18:59:33.846164 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.846142    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4afd685c-067a-40ce-becd-80a838afb556-maas-api-xks-ca\") pod \"maas-api-revoke-sub-045c23ec-5s4dt\" (UID: \"4afd685c-067a-40ce-becd-80a838afb556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045c23ec-5s4dt"
Oct 30 18:59:33.898794 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.898769    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkgdm\" (UniqueName: \"kubernetes.io/projected/4afd685c-067a-40ce-becd-80a838afb556-kube-api-access-kkgdm\") pod \"maas-api-revoke-sub-045c23ec-5s4dt\" (UID: \"4afd685c-067a-40ce-becd-80a838afb556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045c23ec-5s4dt"
Oct 30 18:59:33.963368 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:33.963341    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-045c23ec-5s4dt"
Oct 30 18:59:34.065946 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:34.062324    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:34.356829 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:59:34.356796    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4afd685c_067a_40ce_becd_80a838afb556.slice/crio-ed3635dfc7a169bbb6853eaceb077c34c3f3235264044bf63efc28400e6e612a WatchSource:0}: Error finding container ed3635dfc7a169bbb6853eaceb077c34c3f3235264044bf63efc28400e6e612a: Status 404 returned error can't find the container with id ed3635dfc7a169bbb6853eaceb077c34c3f3235264044bf63efc28400e6e612a
Oct 30 18:59:34.357813 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:34.357787    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-045c23ec-5s4dt"]
Oct 30 18:59:34.494209 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:34.494175    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-045c23ec-5s4dt" event={"ID":"4afd685c-067a-40ce-becd-80a838afb556","Type":"ContainerStarted","Data":"ed3635dfc7a169bbb6853eaceb077c34c3f3235264044bf63efc28400e6e612a"}
Oct 30 18:59:34.495662 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:34.495642    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65470d5-wbdgk" event={"ID":"4b428a6a-2a07-4b92-89cf-4646ed10f619","Type":"ContainerDied","Data":"36fdb470165b850117707d8a86a5d40026e1eefc852deeeabb252ce91637b0e2"}
Oct 30 18:59:34.495757 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:34.495677    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36fdb470165b850117707d8a86a5d40026e1eefc852deeeabb252ce91637b0e2"
Oct 30 18:59:34.495757 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:34.495689    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65470d5-wbdgk"
Oct 30 18:59:35.518089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:35.517993    4924 generic.go:356] "Generic (PLEG): container finished" podID="4afd685c-067a-40ce-becd-80a838afb556" containerID="2f69f6399b814ebdd088052e7553c28f48565f9463081f4d74c89ea1e08e1183" exitCode=0
Oct 30 18:59:35.518517 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:35.518082    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-045c23ec-5s4dt" event={"ID":"4afd685c-067a-40ce-becd-80a838afb556","Type":"ContainerDied","Data":"2f69f6399b814ebdd088052e7553c28f48565f9463081f4d74c89ea1e08e1183"}
Oct 30 18:59:35.941117 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:35.941084    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:36.690343 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:36.690319    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-045c23ec-5s4dt"
Oct 30 18:59:36.782418 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:36.782342    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4afd685c-067a-40ce-becd-80a838afb556-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4afd685c-067a-40ce-becd-80a838afb556-maas-api-openshift-ca\") pod \"4afd685c-067a-40ce-becd-80a838afb556\" (UID: \"4afd685c-067a-40ce-becd-80a838afb556\") "
Oct 30 18:59:36.782563 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:36.782456    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4afd685c-067a-40ce-becd-80a838afb556-kube-api-access-kkgdm\" (UniqueName: \"kubernetes.io/projected/4afd685c-067a-40ce-becd-80a838afb556-kube-api-access-kkgdm\") pod \"4afd685c-067a-40ce-becd-80a838afb556\" (UID: \"4afd685c-067a-40ce-becd-80a838afb556\") "
Oct 30 18:59:36.782563 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:36.782502    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4afd685c-067a-40ce-becd-80a838afb556-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4afd685c-067a-40ce-becd-80a838afb556-maas-api-xks-ca\") pod \"4afd685c-067a-40ce-becd-80a838afb556\" (UID: \"4afd685c-067a-40ce-becd-80a838afb556\") "
Oct 30 18:59:36.782703 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:36.782677    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4afd685c-067a-40ce-becd-80a838afb556-maas-api-openshift-ca" pod "4afd685c-067a-40ce-becd-80a838afb556" (UID: "4afd685c-067a-40ce-becd-80a838afb556"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:36.782939 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:36.782926    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4afd685c-067a-40ce-becd-80a838afb556-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:36.784530 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:36.784511    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4afd685c-067a-40ce-becd-80a838afb556-kube-api-access-kkgdm" pod "4afd685c-067a-40ce-becd-80a838afb556" (UID: "4afd685c-067a-40ce-becd-80a838afb556"). InnerVolumeSpecName "kube-api-access-kkgdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:36.784619 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:36.784581    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4afd685c-067a-40ce-becd-80a838afb556-maas-api-xks-ca" pod "4afd685c-067a-40ce-becd-80a838afb556" (UID: "4afd685c-067a-40ce-becd-80a838afb556"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:36.884500 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:36.884464    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkgdm\" (UniqueName: \"kubernetes.io/projected/4afd685c-067a-40ce-becd-80a838afb556-kube-api-access-kkgdm\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:36.884500 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:36.884496    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4afd685c-067a-40ce-becd-80a838afb556-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:37.533998 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:37.533969    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-045c23ec-5s4dt"
Oct 30 18:59:37.534198 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:37.534005    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-045c23ec-5s4dt" event={"ID":"4afd685c-067a-40ce-becd-80a838afb556","Type":"ContainerDied","Data":"ed3635dfc7a169bbb6853eaceb077c34c3f3235264044bf63efc28400e6e612a"}
Oct 30 18:59:37.534198 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:37.534051    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed3635dfc7a169bbb6853eaceb077c34c3f3235264044bf63efc28400e6e612a"
Oct 30 18:59:38.910908 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:38.910852    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:39.028102 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:39.028045    4924 scope.go:122] "RemoveContainer" containerID="ecc75a578750f62c0c46f2ec8ffb810417dcb781f3cf1659da5ef61fa574e513"
Oct 30 18:59:40.054519 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:40.054488    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5be1def-gm8zk"]
Oct 30 18:59:40.100879 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:40.100851    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5be1def-gm8zk"]
Oct 30 18:59:40.835124 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:40.835086    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51274534-a4ac-4b78-9419-707c67528744" path="/var/lib/kubelet/pods/51274534-a4ac-4b78-9419-707c67528744/volumes"
Oct 30 18:59:41.069519 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:41.069490    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f478c53e-wsxp2"]
Oct 30 18:59:41.115523 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:41.115498    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f478c53e-wsxp2"]
Oct 30 18:59:42.836417 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:42.836383    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26c96a23-72c8-44c6-8d2b-62d85d750bb4" path="/var/lib/kubelet/pods/26c96a23-72c8-44c6-8d2b-62d85d750bb4/volumes"
Oct 30 18:59:43.111608 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:43.111578    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dfda3bc-9krtl"]
Oct 30 18:59:43.158563 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:43.158538    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dfda3bc-9krtl"]
Oct 30 18:59:44.052714 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:44.052679    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e766e00b-jhpgd"]
Oct 30 18:59:44.056704 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:44.056676    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e766e00b-jhpgd"]
Oct 30 18:59:44.120620 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:44.120588    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:44.836377 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:44.836343    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11a916f3-9b9b-4e2e-85d9-e19f0e14a355" path="/var/lib/kubelet/pods/11a916f3-9b9b-4e2e-85d9-e19f0e14a355/volumes"
Oct 30 18:59:44.836808 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:44.836795    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a0e3b75-0597-4169-9dfa-848c4359eec3" path="/var/lib/kubelet/pods/1a0e3b75-0597-4169-9dfa-848c4359eec3/volumes"
Oct 30 18:59:45.086539 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:45.086462    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49d82f15-8szzf"]
Oct 30 18:59:45.087315 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:45.087296    4924 cpu_manager.go:414] "containerMap: removing container" podUID="4afd685c-067a-40ce-becd-80a838afb556" containerName="revoke-keys"
Oct 30 18:59:45.087315 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:45.087316    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4afd685c-067a-40ce-becd-80a838afb556" containerName="revoke-keys"
Oct 30 18:59:45.087455 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:45.087446    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="4afd685c-067a-40ce-becd-80a838afb556" containerName="revoke-keys"
Oct 30 18:59:45.092512 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:45.092494    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d82f15-8szzf"
Oct 30 18:59:45.144381 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:45.144354    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49d82f15-8szzf"]
Oct 30 18:59:45.185534 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:45.185501    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 18:59:45.185731 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:45.185591    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 18:59:45.185731 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:45.185699    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 18:59:45.188236 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:45.188211    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 18:59:45.192530 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:45.192507    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2sfp\" (UniqueName: \"kubernetes.io/projected/9a0852d7-de91-4525-b8ad-e4a34abf76a8-kube-api-access-j2sfp\") pod \"maas-api-revoke-sub-49d82f15-8szzf\" (UID: \"9a0852d7-de91-4525-b8ad-e4a34abf76a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d82f15-8szzf"
Oct 30 18:59:45.192657 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:45.192633    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a0852d7-de91-4525-b8ad-e4a34abf76a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49d82f15-8szzf\" (UID: \"9a0852d7-de91-4525-b8ad-e4a34abf76a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d82f15-8szzf"
Oct 30 18:59:45.192811 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:45.192786    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a0852d7-de91-4525-b8ad-e4a34abf76a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49d82f15-8szzf\" (UID: \"9a0852d7-de91-4525-b8ad-e4a34abf76a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d82f15-8szzf"
Oct 30 18:59:45.294551 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:45.294516    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2sfp\" (UniqueName: \"kubernetes.io/projected/9a0852d7-de91-4525-b8ad-e4a34abf76a8-kube-api-access-j2sfp\") pod \"maas-api-revoke-sub-49d82f15-8szzf\" (UID: \"9a0852d7-de91-4525-b8ad-e4a34abf76a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d82f15-8szzf"
Oct 30 18:59:45.294720 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:45.294609    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a0852d7-de91-4525-b8ad-e4a34abf76a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49d82f15-8szzf\" (UID: \"9a0852d7-de91-4525-b8ad-e4a34abf76a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d82f15-8szzf"
Oct 30 18:59:45.294790 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:45.294747    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a0852d7-de91-4525-b8ad-e4a34abf76a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49d82f15-8szzf\" (UID: \"9a0852d7-de91-4525-b8ad-e4a34abf76a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d82f15-8szzf"
Oct 30 18:59:45.295390 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:45.295368    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a0852d7-de91-4525-b8ad-e4a34abf76a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49d82f15-8szzf\" (UID: \"9a0852d7-de91-4525-b8ad-e4a34abf76a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d82f15-8szzf"
Oct 30 18:59:45.297129 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:45.297107    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a0852d7-de91-4525-b8ad-e4a34abf76a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49d82f15-8szzf\" (UID: \"9a0852d7-de91-4525-b8ad-e4a34abf76a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d82f15-8szzf"
Oct 30 18:59:45.354755 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:45.354726    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2sfp\" (UniqueName: \"kubernetes.io/projected/9a0852d7-de91-4525-b8ad-e4a34abf76a8-kube-api-access-j2sfp\") pod \"maas-api-revoke-sub-49d82f15-8szzf\" (UID: \"9a0852d7-de91-4525-b8ad-e4a34abf76a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d82f15-8szzf"
Oct 30 18:59:45.405980 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:45.405949    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d82f15-8szzf"
Oct 30 18:59:45.711861 ip-10-0-131-224 kubenswrapper[4924]: E1030 18:59:45.711775    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 18:59:45.777312 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:59:45.777270    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a0852d7_de91_4525_b8ad_e4a34abf76a8.slice/crio-ac2ed31a0e3c1d383e4f9947779ab0291ab197c056258a8cbdd728ee61766a58 WatchSource:0}: Error finding container ac2ed31a0e3c1d383e4f9947779ab0291ab197c056258a8cbdd728ee61766a58: Status 404 returned error can't find the container with id ac2ed31a0e3c1d383e4f9947779ab0291ab197c056258a8cbdd728ee61766a58
Oct 30 18:59:45.778852 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:45.778754    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49d82f15-8szzf"]
Oct 30 18:59:46.606918 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:46.606818    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d82f15-8szzf" event={"ID":"9a0852d7-de91-4525-b8ad-e4a34abf76a8","Type":"ContainerStarted","Data":"1bf571168ccdcb044f03f5555781cb8a22171b72213d30611995c250c7f2dc0d"}
Oct 30 18:59:46.606918 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:46.606881    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d82f15-8szzf" event={"ID":"9a0852d7-de91-4525-b8ad-e4a34abf76a8","Type":"ContainerStarted","Data":"ac2ed31a0e3c1d383e4f9947779ab0291ab197c056258a8cbdd728ee61766a58"}
Oct 30 18:59:46.720233 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:46.720188    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d82f15-8szzf" podStartSLOduration=1.720174595 podStartE2EDuration="1.720174595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:46.717176885 +0000 UTC m=+1387.994227330" watchObservedRunningTime="2026-10-30 18:59:46.720174595 +0000 UTC m=+1387.997225075"
Oct 30 18:59:47.618184 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:47.618153    4924 generic.go:356] "Generic (PLEG): container finished" podID="9a0852d7-de91-4525-b8ad-e4a34abf76a8" containerID="1bf571168ccdcb044f03f5555781cb8a22171b72213d30611995c250c7f2dc0d" exitCode=0
Oct 30 18:59:47.618642 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:47.618244    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d82f15-8szzf" event={"ID":"9a0852d7-de91-4525-b8ad-e4a34abf76a8","Type":"ContainerDied","Data":"1bf571168ccdcb044f03f5555781cb8a22171b72213d30611995c250c7f2dc0d"}
Oct 30 18:59:49.076876 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:49.076770    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5a84620-d7lxx"]
Oct 30 18:59:49.155089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:49.143719    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5a84620-d7lxx"]
Oct 30 18:59:49.155089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:49.143836    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a84620-d7lxx"
Oct 30 18:59:49.369144 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:49.368985    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c613a857-c27b-4cde-a044-a90005f8f2a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5a84620-d7lxx\" (UID: \"c613a857-c27b-4cde-a044-a90005f8f2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a84620-d7lxx"
Oct 30 18:59:49.369144 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:49.369129    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5d2t\" (UniqueName: \"kubernetes.io/projected/c613a857-c27b-4cde-a044-a90005f8f2a3-kube-api-access-h5d2t\") pod \"maas-api-revoke-sub-d5a84620-d7lxx\" (UID: \"c613a857-c27b-4cde-a044-a90005f8f2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a84620-d7lxx"
Oct 30 18:59:49.369385 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:49.369243    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c613a857-c27b-4cde-a044-a90005f8f2a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5a84620-d7lxx\" (UID: \"c613a857-c27b-4cde-a044-a90005f8f2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a84620-d7lxx"
Oct 30 18:59:49.491088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:49.481821    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5d2t\" (UniqueName: \"kubernetes.io/projected/c613a857-c27b-4cde-a044-a90005f8f2a3-kube-api-access-h5d2t\") pod \"maas-api-revoke-sub-d5a84620-d7lxx\" (UID: \"c613a857-c27b-4cde-a044-a90005f8f2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a84620-d7lxx"
Oct 30 18:59:49.491088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:49.481937    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c613a857-c27b-4cde-a044-a90005f8f2a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5a84620-d7lxx\" (UID: \"c613a857-c27b-4cde-a044-a90005f8f2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a84620-d7lxx"
Oct 30 18:59:49.491088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:49.482102    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c613a857-c27b-4cde-a044-a90005f8f2a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5a84620-d7lxx\" (UID: \"c613a857-c27b-4cde-a044-a90005f8f2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a84620-d7lxx"
Oct 30 18:59:49.491088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:49.483105    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c613a857-c27b-4cde-a044-a90005f8f2a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5a84620-d7lxx\" (UID: \"c613a857-c27b-4cde-a044-a90005f8f2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a84620-d7lxx"
Oct 30 18:59:49.522699 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:49.522662    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c613a857-c27b-4cde-a044-a90005f8f2a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5a84620-d7lxx\" (UID: \"c613a857-c27b-4cde-a044-a90005f8f2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a84620-d7lxx"
Oct 30 18:59:49.560577 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:49.560543    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5d2t\" (UniqueName: \"kubernetes.io/projected/c613a857-c27b-4cde-a044-a90005f8f2a3-kube-api-access-h5d2t\") pod \"maas-api-revoke-sub-d5a84620-d7lxx\" (UID: \"c613a857-c27b-4cde-a044-a90005f8f2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a84620-d7lxx"
Oct 30 18:59:49.667396 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:49.667311    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d82f15-8szzf"
Oct 30 18:59:49.688054 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:49.688014    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a0852d7-de91-4525-b8ad-e4a34abf76a8-kube-api-access-j2sfp\" (UniqueName: \"kubernetes.io/projected/9a0852d7-de91-4525-b8ad-e4a34abf76a8-kube-api-access-j2sfp\") pod \"9a0852d7-de91-4525-b8ad-e4a34abf76a8\" (UID: \"9a0852d7-de91-4525-b8ad-e4a34abf76a8\") "
Oct 30 18:59:49.700093 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:49.695778    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a0852d7-de91-4525-b8ad-e4a34abf76a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a0852d7-de91-4525-b8ad-e4a34abf76a8-maas-api-xks-ca\") pod \"9a0852d7-de91-4525-b8ad-e4a34abf76a8\" (UID: \"9a0852d7-de91-4525-b8ad-e4a34abf76a8\") "
Oct 30 18:59:49.710323 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:49.710230    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a0852d7-de91-4525-b8ad-e4a34abf76a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a0852d7-de91-4525-b8ad-e4a34abf76a8-maas-api-openshift-ca\") pod \"9a0852d7-de91-4525-b8ad-e4a34abf76a8\" (UID: \"9a0852d7-de91-4525-b8ad-e4a34abf76a8\") "
Oct 30 18:59:49.714432 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:49.714295    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a0852d7-de91-4525-b8ad-e4a34abf76a8-maas-api-openshift-ca" pod "9a0852d7-de91-4525-b8ad-e4a34abf76a8" (UID: "9a0852d7-de91-4525-b8ad-e4a34abf76a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:49.730091 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:49.723673    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a0852d7-de91-4525-b8ad-e4a34abf76a8-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:49.734872 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:49.734846    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a0852d7-de91-4525-b8ad-e4a34abf76a8-kube-api-access-j2sfp" pod "9a0852d7-de91-4525-b8ad-e4a34abf76a8" (UID: "9a0852d7-de91-4525-b8ad-e4a34abf76a8"). InnerVolumeSpecName "kube-api-access-j2sfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:49.735612 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:49.735582    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:49.754244 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:49.748943    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d82f15-8szzf" event={"ID":"9a0852d7-de91-4525-b8ad-e4a34abf76a8","Type":"ContainerDied","Data":"ac2ed31a0e3c1d383e4f9947779ab0291ab197c056258a8cbdd728ee61766a58"}
Oct 30 18:59:49.754508 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:49.754474    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac2ed31a0e3c1d383e4f9947779ab0291ab197c056258a8cbdd728ee61766a58"
Oct 30 18:59:49.754646 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:49.749015    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49d82f15-8szzf"
Oct 30 18:59:49.858253 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:49.858213    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2sfp\" (UniqueName: \"kubernetes.io/projected/9a0852d7-de91-4525-b8ad-e4a34abf76a8-kube-api-access-j2sfp\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:49.859452 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:49.859426    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a84620-d7lxx"
Oct 30 18:59:49.910105 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:49.901996    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l" podUID="bdc8df4b-8c94-40b8-b4be-80d1432d87de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:50.195764 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:50.195703    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj" podUID="51f3e639-9cd4-4aaa-a568-f19b6d2704fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:50.212305 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:50.212265    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a0852d7-de91-4525-b8ad-e4a34abf76a8-maas-api-xks-ca" pod "9a0852d7-de91-4525-b8ad-e4a34abf76a8" (UID: "9a0852d7-de91-4525-b8ad-e4a34abf76a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:50.280755 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:50.280152    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" podUID="eb82dd7d-e682-42a2-924e-a84f21172fc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:50.294703 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:50.294445    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86" podUID="60a58597-fdc4-4722-8d2a-29e475f84a76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:50.315140 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:50.314170    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m" podUID="1a90412d-c047-422e-b3c7-9546d286c303" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:50.338297 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:50.316717    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a0852d7-de91-4525-b8ad-e4a34abf76a8-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:50.338297 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:50.325140    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2" podUID="fdf4711c-418a-436b-848a-d46974566573" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:50.338297 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:50.337140    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd" podUID="03ac954a-d07b-4523-8d5a-a13dffd0bd2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:50.367572 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:50.356471    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5" podUID="925a4de9-b451-4263-9a1c-eebbbae4a08c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:50.551089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:50.546340    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7" podUID="97a3b018-9083-4b7c-af88-6edddc585d67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:50.606560 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:50.606519    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt" podUID="955e98b9-864b-4636-9511-8b69f981eaa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:50.915088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:50.910191    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l" podUID="bdc8df4b-8c94-40b8-b4be-80d1432d87de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:50.967583 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:50.966164    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd" podUID="a3d792df-177d-4f7a-bba8-f6351175462e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:50.987089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:50.982093    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld" podUID="f04af09f-b8d2-4b47-b0b3-88eb12adff93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:51.062101 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:51.039320    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg" podUID="9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:51.130089 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:51.111665    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:59:51.400600 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:51.400554    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd" podUID="03ac954a-d07b-4523-8d5a-a13dffd0bd2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 18:59:51.591175 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:51.479810    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5" podUID="925a4de9-b451-4263-9a1c-eebbbae4a08c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 18:59:51.591175 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:51.480578    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" podUID="eb82dd7d-e682-42a2-924e-a84f21172fc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 30 18:59:51.591175 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:51.480827    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86" podUID="60a58597-fdc4-4722-8d2a-29e475f84a76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 18:59:51.591175 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:51.480955    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:59:51.591175 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:51.481014    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:59:51.591175 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:51.481050    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:59:51.591175 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:51.481104    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:59:51.591175 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:51.486490    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m" podUID="1a90412d-c047-422e-b3c7-9546d286c303" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 18:59:51.591175 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:51.486752    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2" podUID="fdf4711c-418a-436b-848a-d46974566573" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:51.591175 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:51.487770    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:59:51.591175 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:51.487821    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:59:51.591175 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:51.507593    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj" podUID="51f3e639-9cd4-4aaa-a568-f19b6d2704fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 18:59:51.591175 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:51.507925    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:59:51.591175 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:51.557824    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7" podUID="97a3b018-9083-4b7c-af88-6edddc585d67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:51.591175 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:51.558236    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:59:51.613094 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:51.607156    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt" podUID="955e98b9-864b-4636-9511-8b69f981eaa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:51.613094 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:51.607618    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:59:52.158084 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:51.983983    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld" podUID="f04af09f-b8d2-4b47-b0b3-88eb12adff93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:52.158084 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:52.060466    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l" podUID="bdc8df4b-8c94-40b8-b4be-80d1432d87de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Oct 30 18:59:52.158084 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:52.066882    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg" podUID="9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Oct 30 18:59:52.191300 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:52.191258    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:59:52.191634 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:52.191613    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:59:52.682635 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:52.682592    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5" podUID="925a4de9-b451-4263-9a1c-eebbbae4a08c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 18:59:53.538146 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:52.738117    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7" podUID="97a3b018-9083-4b7c-af88-6edddc585d67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 18:59:53.538146 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:52.761851    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj" podUID="51f3e639-9cd4-4aaa-a568-f19b6d2704fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:53.538146 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:52.762724    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86" podUID="60a58597-fdc4-4722-8d2a-29e475f84a76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 18:59:53.538146 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:52.797418    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m" podUID="1a90412d-c047-422e-b3c7-9546d286c303" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 18:59:53.538146 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:52.798745    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd" podUID="a3d792df-177d-4f7a-bba8-f6351175462e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded"
Oct 30 18:59:53.566090 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:53.542520    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5a84620-d7lxx"]
Oct 30 18:59:53.602120 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:53.598317    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:59:53.760094 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:53.711855    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-901b4303-2rvkf"]
Oct 30 18:59:53.801680 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:53.801648    4924 cpu_manager.go:414] "containerMap: removing container" podUID="9a0852d7-de91-4525-b8ad-e4a34abf76a8" containerName="revoke-keys"
Oct 30 18:59:53.801862 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:53.801847    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a0852d7-de91-4525-b8ad-e4a34abf76a8" containerName="revoke-keys"
Oct 30 18:59:53.802238 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:53.802221    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a0852d7-de91-4525-b8ad-e4a34abf76a8" containerName="revoke-keys"
Oct 30 18:59:53.817010 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:53.816971    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7" podUID="97a3b018-9083-4b7c-af88-6edddc585d67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded"
Oct 30 18:59:53.817274 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:53.188235    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg" podUID="9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Oct 30 18:59:53.842156 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:53.095863    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld" podUID="f04af09f-b8d2-4b47-b0b3-88eb12adff93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": context deadline exceeded"
Oct 30 18:59:53.842408 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:52.798766    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd" podUID="03ac954a-d07b-4523-8d5a-a13dffd0bd2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 18:59:53.842839 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:53.017861    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd" podUID="03ac954a-d07b-4523-8d5a-a13dffd0bd2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded"
Oct 30 18:59:53.842990 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:52.759228    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" podUID="eb82dd7d-e682-42a2-924e-a84f21172fc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 30 18:59:53.843420 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:53.021091    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld" podUID="f04af09f-b8d2-4b47-b0b3-88eb12adff93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:53.843815 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:53.069203    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt" podUID="955e98b9-864b-4636-9511-8b69f981eaa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:53.844226 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:52.801849    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2" podUID="fdf4711c-418a-436b-848a-d46974566573" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 18:59:53.898522 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:53.229002    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l" podUID="bdc8df4b-8c94-40b8-b4be-80d1432d87de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Oct 30 18:59:53.975148 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:53.967545    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-901b4303-2rvkf"
Oct 30 18:59:54.037170 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.036500    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd" podUID="a3d792df-177d-4f7a-bba8-f6351175462e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:54.187447 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.187410    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 18:59:54.210629 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.190606    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 18:59:54.246991 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.246903    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a84620-d7lxx" event={"ID":"c613a857-c27b-4cde-a044-a90005f8f2a3","Type":"ContainerStarted","Data":"0a326be9e9a28f3bb24990d71978176104f5a0fcb63f3efa93208ff7d1c41e43"}
Oct 30 18:59:54.249345 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.249317    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-901b4303-2rvkf"]
Oct 30 18:59:54.253675 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.253651    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06839456-517f-4fd9-b469-bb30d10c0e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-901b4303-2rvkf\" (UID: \"06839456-517f-4fd9-b469-bb30d10c0e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901b4303-2rvkf"
Oct 30 18:59:54.253998 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.253976    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06839456-517f-4fd9-b469-bb30d10c0e5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-901b4303-2rvkf\" (UID: \"06839456-517f-4fd9-b469-bb30d10c0e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901b4303-2rvkf"
Oct 30 18:59:54.254159 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.254143    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbjt7\" (UniqueName: \"kubernetes.io/projected/06839456-517f-4fd9-b469-bb30d10c0e5e-kube-api-access-zbjt7\") pod \"maas-api-revoke-sub-901b4303-2rvkf\" (UID: \"06839456-517f-4fd9-b469-bb30d10c0e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901b4303-2rvkf"
Oct 30 18:59:54.388088 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.371356    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06839456-517f-4fd9-b469-bb30d10c0e5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-901b4303-2rvkf\" (UID: \"06839456-517f-4fd9-b469-bb30d10c0e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901b4303-2rvkf"
Oct 30 18:59:54.416719 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.416687    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbjt7\" (UniqueName: \"kubernetes.io/projected/06839456-517f-4fd9-b469-bb30d10c0e5e-kube-api-access-zbjt7\") pod \"maas-api-revoke-sub-901b4303-2rvkf\" (UID: \"06839456-517f-4fd9-b469-bb30d10c0e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901b4303-2rvkf"
Oct 30 18:59:54.417339 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.417317    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06839456-517f-4fd9-b469-bb30d10c0e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-901b4303-2rvkf\" (UID: \"06839456-517f-4fd9-b469-bb30d10c0e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901b4303-2rvkf"
Oct 30 18:59:54.418728 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.418708    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06839456-517f-4fd9-b469-bb30d10c0e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-901b4303-2rvkf\" (UID: \"06839456-517f-4fd9-b469-bb30d10c0e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901b4303-2rvkf"
Oct 30 18:59:54.475287 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.475240    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt" podUID="955e98b9-864b-4636-9511-8b69f981eaa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": context deadline exceeded"
Oct 30 18:59:54.546197 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.538429    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd" podUID="a3d792df-177d-4f7a-bba8-f6351175462e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:54.570097 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.538757    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj" podUID="51f3e639-9cd4-4aaa-a568-f19b6d2704fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 18:59:54.570097 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.539883    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5" podUID="925a4de9-b451-4263-9a1c-eebbbae4a08c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 18:59:54.570097 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.539912    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7" podUID="97a3b018-9083-4b7c-af88-6edddc585d67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 18:59:54.570097 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.539941    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86" podUID="60a58597-fdc4-4722-8d2a-29e475f84a76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:54.570097 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.567941    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbjt7\" (UniqueName: \"kubernetes.io/projected/06839456-517f-4fd9-b469-bb30d10c0e5e-kube-api-access-zbjt7\") pod \"maas-api-revoke-sub-901b4303-2rvkf\" (UID: \"06839456-517f-4fd9-b469-bb30d10c0e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901b4303-2rvkf"
Oct 30 18:59:54.674031 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.674000    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 18:59:54.789612 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.789581    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 18:59:54.795135 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.795102    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5" podUID="925a4de9-b451-4263-9a1c-eebbbae4a08c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:54.797051 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.796985    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 18:59:54.846602 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.846562    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt" podUID="955e98b9-864b-4636-9511-8b69f981eaa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:54.847327 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.847297    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld" podUID="f04af09f-b8d2-4b47-b0b3-88eb12adff93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 18:59:54.851389 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.843567    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" podUID="eb82dd7d-e682-42a2-924e-a84f21172fc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:54.855567 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.831155    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg" podUID="9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:54.933729 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.933674    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06839456-517f-4fd9-b469-bb30d10c0e5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-901b4303-2rvkf\" (UID: \"06839456-517f-4fd9-b469-bb30d10c0e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-901b4303-2rvkf"
Oct 30 18:59:54.951607 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.951549    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l" podUID="bdc8df4b-8c94-40b8-b4be-80d1432d87de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 18:59:54.969960 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.958351    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 18:59:54.969960 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.958443    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 18:59:54.969960 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.958973    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 18:59:54.995352 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.995151    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 18:59:54.995525 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.995475    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 18:59:54.996242 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:54.996221    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 18:59:55.000877 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:55.000259    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 18:59:55.015583 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:55.015558    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 18:59:55.087506 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:55.087428    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-901b4303-2rvkf"
Oct 30 18:59:55.241690 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:55.241149    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a84620-d7lxx" event={"ID":"c613a857-c27b-4cde-a044-a90005f8f2a3","Type":"ContainerStarted","Data":"4937402652d37cc13ed384b6f0f927046b9a277ffb387470dd75b9492536ab33"}
Oct 30 18:59:55.490922 ip-10-0-131-224 kubenswrapper[4924]: W1030 18:59:55.490884    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06839456_517f_4fd9_b469_bb30d10c0e5e.slice/crio-c9d0638608790843053ed0ee1009b8b13eb3316aeb47d7872ed115f093e5d719 WatchSource:0}: Error finding container c9d0638608790843053ed0ee1009b8b13eb3316aeb47d7872ed115f093e5d719: Status 404 returned error can't find the container with id c9d0638608790843053ed0ee1009b8b13eb3316aeb47d7872ed115f093e5d719
Oct 30 18:59:55.492400 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:55.492372    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-901b4303-2rvkf"]
Oct 30 18:59:55.652264 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:55.652209    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a84620-d7lxx" podStartSLOduration=6.652196192 podStartE2EDuration="6.652196192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:55.648569585 +0000 UTC m=+1396.925620031" watchObservedRunningTime="2026-10-30 18:59:55.652196192 +0000 UTC m=+1396.929246636"
Oct 30 18:59:56.049846 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:56.049747    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f752cb5-hpwb4"]
Oct 30 18:59:56.054320 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:56.054289    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f752cb5-hpwb4"]
Oct 30 18:59:56.250505 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:56.250454    4924 generic.go:356] "Generic (PLEG): container finished" podID="c613a857-c27b-4cde-a044-a90005f8f2a3" containerID="4937402652d37cc13ed384b6f0f927046b9a277ffb387470dd75b9492536ab33" exitCode=0
Oct 30 18:59:56.250672 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:56.250542    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a84620-d7lxx" event={"ID":"c613a857-c27b-4cde-a044-a90005f8f2a3","Type":"ContainerDied","Data":"4937402652d37cc13ed384b6f0f927046b9a277ffb387470dd75b9492536ab33"}
Oct 30 18:59:56.252558 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:56.252534    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-901b4303-2rvkf" event={"ID":"06839456-517f-4fd9-b469-bb30d10c0e5e","Type":"ContainerStarted","Data":"3b3a02295eaadb323161ea4e7a02e3ab549df6ec3b0130a441226ee632370885"}
Oct 30 18:59:56.252664 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:56.252578    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-901b4303-2rvkf" event={"ID":"06839456-517f-4fd9-b469-bb30d10c0e5e","Type":"ContainerStarted","Data":"c9d0638608790843053ed0ee1009b8b13eb3316aeb47d7872ed115f093e5d719"}
Oct 30 18:59:56.836918 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:56.836841    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c05fc13a-ba76-459d-aa9b-cf06ba7ece60" path="/var/lib/kubelet/pods/c05fc13a-ba76-459d-aa9b-cf06ba7ece60/volumes"
Oct 30 18:59:57.011888 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:57.011837    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-901b4303-2rvkf" podStartSLOduration=5.011824529 podStartE2EDuration="5.011824529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:57.010599316 +0000 UTC m=+1398.287649761" watchObservedRunningTime="2026-10-30 18:59:57.011824529 +0000 UTC m=+1398.288874975"
Oct 30 18:59:57.096218 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:57.096118    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 18:59:57.262165 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:57.262133    4924 generic.go:356] "Generic (PLEG): container finished" podID="06839456-517f-4fd9-b469-bb30d10c0e5e" containerID="3b3a02295eaadb323161ea4e7a02e3ab549df6ec3b0130a441226ee632370885" exitCode=0
Oct 30 18:59:57.262332 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:57.262213    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-901b4303-2rvkf" event={"ID":"06839456-517f-4fd9-b469-bb30d10c0e5e","Type":"ContainerDied","Data":"3b3a02295eaadb323161ea4e7a02e3ab549df6ec3b0130a441226ee632370885"}
Oct 30 18:59:57.430785 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:57.430757    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a84620-d7lxx"
Oct 30 18:59:57.595580 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:57.595548    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c613a857-c27b-4cde-a044-a90005f8f2a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c613a857-c27b-4cde-a044-a90005f8f2a3-maas-api-openshift-ca\") pod \"c613a857-c27b-4cde-a044-a90005f8f2a3\" (UID: \"c613a857-c27b-4cde-a044-a90005f8f2a3\") "
Oct 30 18:59:57.595765 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:57.595600    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c613a857-c27b-4cde-a044-a90005f8f2a3-kube-api-access-h5d2t\" (UniqueName: \"kubernetes.io/projected/c613a857-c27b-4cde-a044-a90005f8f2a3-kube-api-access-h5d2t\") pod \"c613a857-c27b-4cde-a044-a90005f8f2a3\" (UID: \"c613a857-c27b-4cde-a044-a90005f8f2a3\") "
Oct 30 18:59:57.595765 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:57.595676    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c613a857-c27b-4cde-a044-a90005f8f2a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c613a857-c27b-4cde-a044-a90005f8f2a3-maas-api-xks-ca\") pod \"c613a857-c27b-4cde-a044-a90005f8f2a3\" (UID: \"c613a857-c27b-4cde-a044-a90005f8f2a3\") "
Oct 30 18:59:57.596048 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:57.596009    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c613a857-c27b-4cde-a044-a90005f8f2a3-maas-api-openshift-ca" pod "c613a857-c27b-4cde-a044-a90005f8f2a3" (UID: "c613a857-c27b-4cde-a044-a90005f8f2a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:57.596434 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:57.596414    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c613a857-c27b-4cde-a044-a90005f8f2a3-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:57.597903 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:57.597877    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c613a857-c27b-4cde-a044-a90005f8f2a3-kube-api-access-h5d2t" pod "c613a857-c27b-4cde-a044-a90005f8f2a3" (UID: "c613a857-c27b-4cde-a044-a90005f8f2a3"). InnerVolumeSpecName "kube-api-access-h5d2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:57.597903 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:57.597881    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c613a857-c27b-4cde-a044-a90005f8f2a3-maas-api-xks-ca" pod "c613a857-c27b-4cde-a044-a90005f8f2a3" (UID: "c613a857-c27b-4cde-a044-a90005f8f2a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:57.698007 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:57.697928    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5d2t\" (UniqueName: \"kubernetes.io/projected/c613a857-c27b-4cde-a044-a90005f8f2a3-kube-api-access-h5d2t\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:57.698007 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:57.697962    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c613a857-c27b-4cde-a044-a90005f8f2a3-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:58.056558 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:58.056417    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2adf84ef-vqcrt"]
Oct 30 18:59:58.060615 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:58.060590    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2adf84ef-vqcrt"]
Oct 30 18:59:58.272740 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:58.272705    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a84620-d7lxx"
Oct 30 18:59:58.272740 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:58.272718    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a84620-d7lxx" event={"ID":"c613a857-c27b-4cde-a044-a90005f8f2a3","Type":"ContainerDied","Data":"0a326be9e9a28f3bb24990d71978176104f5a0fcb63f3efa93208ff7d1c41e43"}
Oct 30 18:59:58.273200 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:58.272764    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a326be9e9a28f3bb24990d71978176104f5a0fcb63f3efa93208ff7d1c41e43"
Oct 30 18:59:58.442165 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:58.442141    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-901b4303-2rvkf"
Oct 30 18:59:58.610703 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:58.610668    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06839456-517f-4fd9-b469-bb30d10c0e5e-kube-api-access-zbjt7\" (UniqueName: \"kubernetes.io/projected/06839456-517f-4fd9-b469-bb30d10c0e5e-kube-api-access-zbjt7\") pod \"06839456-517f-4fd9-b469-bb30d10c0e5e\" (UID: \"06839456-517f-4fd9-b469-bb30d10c0e5e\") "
Oct 30 18:59:58.610847 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:58.610766    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06839456-517f-4fd9-b469-bb30d10c0e5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06839456-517f-4fd9-b469-bb30d10c0e5e-maas-api-openshift-ca\") pod \"06839456-517f-4fd9-b469-bb30d10c0e5e\" (UID: \"06839456-517f-4fd9-b469-bb30d10c0e5e\") "
Oct 30 18:59:58.610884 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:58.610854    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06839456-517f-4fd9-b469-bb30d10c0e5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06839456-517f-4fd9-b469-bb30d10c0e5e-maas-api-xks-ca\") pod \"06839456-517f-4fd9-b469-bb30d10c0e5e\" (UID: \"06839456-517f-4fd9-b469-bb30d10c0e5e\") "
Oct 30 18:59:58.611158 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:58.611129    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06839456-517f-4fd9-b469-bb30d10c0e5e-maas-api-openshift-ca" pod "06839456-517f-4fd9-b469-bb30d10c0e5e" (UID: "06839456-517f-4fd9-b469-bb30d10c0e5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:58.611388 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:58.611373    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06839456-517f-4fd9-b469-bb30d10c0e5e-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:58.612860 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:58.612837    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06839456-517f-4fd9-b469-bb30d10c0e5e-maas-api-xks-ca" pod "06839456-517f-4fd9-b469-bb30d10c0e5e" (UID: "06839456-517f-4fd9-b469-bb30d10c0e5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:58.612934 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:58.612902    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06839456-517f-4fd9-b469-bb30d10c0e5e-kube-api-access-zbjt7" pod "06839456-517f-4fd9-b469-bb30d10c0e5e" (UID: "06839456-517f-4fd9-b469-bb30d10c0e5e"). InnerVolumeSpecName "kube-api-access-zbjt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:58.712448 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:58.712423    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06839456-517f-4fd9-b469-bb30d10c0e5e-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:58.712448 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:58.712446    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbjt7\" (UniqueName: \"kubernetes.io/projected/06839456-517f-4fd9-b469-bb30d10c0e5e-kube-api-access-zbjt7\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:58.835233 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:58.835202    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb9ecacf-1867-4a7f-af5b-1b0ab354a66b" path="/var/lib/kubelet/pods/eb9ecacf-1867-4a7f-af5b-1b0ab354a66b/volumes"
Oct 30 18:59:59.282214 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:59.282181    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-901b4303-2rvkf"
Oct 30 18:59:59.282672 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:59.282218    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-901b4303-2rvkf" event={"ID":"06839456-517f-4fd9-b469-bb30d10c0e5e","Type":"ContainerDied","Data":"c9d0638608790843053ed0ee1009b8b13eb3316aeb47d7872ed115f093e5d719"}
Oct 30 18:59:59.282672 ip-10-0-131-224 kubenswrapper[4924]: I1030 18:59:59.282265    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9d0638608790843053ed0ee1009b8b13eb3316aeb47d7872ed115f093e5d719"
Oct 30 19:00:00.186390 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.186351    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl"]
Oct 30 19:00:00.187234 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.187210    4924 cpu_manager.go:414] "containerMap: removing container" podUID="c613a857-c27b-4cde-a044-a90005f8f2a3" containerName="revoke-keys"
Oct 30 19:00:00.187234 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.187232    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c613a857-c27b-4cde-a044-a90005f8f2a3" containerName="revoke-keys"
Oct 30 19:00:00.187419 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.187257    4924 cpu_manager.go:414] "containerMap: removing container" podUID="06839456-517f-4fd9-b469-bb30d10c0e5e" containerName="revoke-keys"
Oct 30 19:00:00.187419 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.187262    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06839456-517f-4fd9-b469-bb30d10c0e5e" containerName="revoke-keys"
Oct 30 19:00:00.187419 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.187366    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="c613a857-c27b-4cde-a044-a90005f8f2a3" containerName="revoke-keys"
Oct 30 19:00:00.187419 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.187382    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="06839456-517f-4fd9-b469-bb30d10c0e5e" containerName="revoke-keys"
Oct 30 19:00:00.191450 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.191418    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576"]
Oct 30 19:00:00.191582 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.191516    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl"
Oct 30 19:00:00.196762 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.196741    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc"]
Oct 30 19:00:00.196873 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.196836    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576"
Oct 30 19:00:00.201208 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.201191    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc"
Oct 30 19:00:00.232423 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.232399    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/884e56e6-4b28-4245-a418-1782703a11df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl\" (UID: \"884e56e6-4b28-4245-a418-1782703a11df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl"
Oct 30 19:00:00.232514 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.232407    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd"]
Oct 30 19:00:00.232514 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.232454    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvktl\" (UniqueName: \"kubernetes.io/projected/884e56e6-4b28-4245-a418-1782703a11df-kube-api-access-lvktl\") pod \"maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl\" (UID: \"884e56e6-4b28-4245-a418-1782703a11df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl"
Oct 30 19:00:00.232603 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.232571    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9q79\" (UniqueName: \"kubernetes.io/projected/1ba14910-fb5d-40ec-bad0-39dfebd1506e-kube-api-access-b9q79\") pod \"maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc\" (UID: \"1ba14910-fb5d-40ec-bad0-39dfebd1506e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc"
Oct 30 19:00:00.232636 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.232603    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ba14910-fb5d-40ec-bad0-39dfebd1506e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc\" (UID: \"1ba14910-fb5d-40ec-bad0-39dfebd1506e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc"
Oct 30 19:00:00.232681 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.232639    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0f224d4-04cc-4493-b313-56626b0f50a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576\" (UID: \"c0f224d4-04cc-4493-b313-56626b0f50a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576"
Oct 30 19:00:00.232790 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.232771    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d22w2\" (UniqueName: \"kubernetes.io/projected/c0f224d4-04cc-4493-b313-56626b0f50a1-kube-api-access-d22w2\") pod \"maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576\" (UID: \"c0f224d4-04cc-4493-b313-56626b0f50a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576"
Oct 30 19:00:00.237440 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.237421    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g"]
Oct 30 19:00:00.237533 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.237520    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd"
Oct 30 19:00:00.240918 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.240900    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:00:00.240997 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.240943    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:00:00.241522 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.241507    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g"
Oct 30 19:00:00.241593 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.241578    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:00:00.280117 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.280087    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl"]
Oct 30 19:00:00.284388 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.284365    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576"]
Oct 30 19:00:00.323895 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.323827    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc"]
Oct 30 19:00:00.328750 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.328718    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd"]
Oct 30 19:00:00.333575 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.333551    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7582ec87-a899-40e7-9fa8-793e9caaf2f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g\" (UID: \"7582ec87-a899-40e7-9fa8-793e9caaf2f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g"
Oct 30 19:00:00.333706 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.333584    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/884e56e6-4b28-4245-a418-1782703a11df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl\" (UID: \"884e56e6-4b28-4245-a418-1782703a11df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl"
Oct 30 19:00:00.333706 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.333610    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvktl\" (UniqueName: \"kubernetes.io/projected/884e56e6-4b28-4245-a418-1782703a11df-kube-api-access-lvktl\") pod \"maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl\" (UID: \"884e56e6-4b28-4245-a418-1782703a11df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl"
Oct 30 19:00:00.333706 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.333638    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56c52368-2c45-4ef9-aa51-d507aafee931-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd\" (UID: \"56c52368-2c45-4ef9-aa51-d507aafee931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd"
Oct 30 19:00:00.333874 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.333857    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9q79\" (UniqueName: \"kubernetes.io/projected/1ba14910-fb5d-40ec-bad0-39dfebd1506e-kube-api-access-b9q79\") pod \"maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc\" (UID: \"1ba14910-fb5d-40ec-bad0-39dfebd1506e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc"
Oct 30 19:00:00.333933 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.333893    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ba14910-fb5d-40ec-bad0-39dfebd1506e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc\" (UID: \"1ba14910-fb5d-40ec-bad0-39dfebd1506e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc"
Oct 30 19:00:00.333985 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.333928    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jxgm\" (UniqueName: \"kubernetes.io/projected/7582ec87-a899-40e7-9fa8-793e9caaf2f2-kube-api-access-2jxgm\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g\" (UID: \"7582ec87-a899-40e7-9fa8-793e9caaf2f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g"
Oct 30 19:00:00.333985 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.333957    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgxdc\" (UniqueName: \"kubernetes.io/projected/56c52368-2c45-4ef9-aa51-d507aafee931-kube-api-access-rgxdc\") pod \"maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd\" (UID: \"56c52368-2c45-4ef9-aa51-d507aafee931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd"
Oct 30 19:00:00.334139 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.334002    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0f224d4-04cc-4493-b313-56626b0f50a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576\" (UID: \"c0f224d4-04cc-4493-b313-56626b0f50a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576"
Oct 30 19:00:00.334203 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.334186    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d22w2\" (UniqueName: \"kubernetes.io/projected/c0f224d4-04cc-4493-b313-56626b0f50a1-kube-api-access-d22w2\") pod \"maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576\" (UID: \"c0f224d4-04cc-4493-b313-56626b0f50a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576"
Oct 30 19:00:00.334585 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.334561    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/884e56e6-4b28-4245-a418-1782703a11df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl\" (UID: \"884e56e6-4b28-4245-a418-1782703a11df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl"
Oct 30 19:00:00.334784 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.334762    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0f224d4-04cc-4493-b313-56626b0f50a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576\" (UID: \"c0f224d4-04cc-4493-b313-56626b0f50a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576"
Oct 30 19:00:00.335000 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.334883    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ba14910-fb5d-40ec-bad0-39dfebd1506e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc\" (UID: \"1ba14910-fb5d-40ec-bad0-39dfebd1506e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc"
Oct 30 19:00:00.386532 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.386505    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g"]
Oct 30 19:00:00.391672 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.391005    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvktl\" (UniqueName: \"kubernetes.io/projected/884e56e6-4b28-4245-a418-1782703a11df-kube-api-access-lvktl\") pod \"maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl\" (UID: \"884e56e6-4b28-4245-a418-1782703a11df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl"
Oct 30 19:00:00.393321 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.393298    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9q79\" (UniqueName: \"kubernetes.io/projected/1ba14910-fb5d-40ec-bad0-39dfebd1506e-kube-api-access-b9q79\") pod \"maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc\" (UID: \"1ba14910-fb5d-40ec-bad0-39dfebd1506e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc"
Oct 30 19:00:00.393571 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.393547    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d22w2\" (UniqueName: \"kubernetes.io/projected/c0f224d4-04cc-4493-b313-56626b0f50a1-kube-api-access-d22w2\") pod \"maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576\" (UID: \"c0f224d4-04cc-4493-b313-56626b0f50a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576"
Oct 30 19:00:00.403649 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.403620    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5215243-8hngd"]
Oct 30 19:00:00.414758 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.414734    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5215243-8hngd"
Oct 30 19:00:00.435236 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.435213    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jxgm\" (UniqueName: \"kubernetes.io/projected/7582ec87-a899-40e7-9fa8-793e9caaf2f2-kube-api-access-2jxgm\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g\" (UID: \"7582ec87-a899-40e7-9fa8-793e9caaf2f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g"
Oct 30 19:00:00.435329 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.435243    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgxdc\" (UniqueName: \"kubernetes.io/projected/56c52368-2c45-4ef9-aa51-d507aafee931-kube-api-access-rgxdc\") pod \"maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd\" (UID: \"56c52368-2c45-4ef9-aa51-d507aafee931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd"
Oct 30 19:00:00.435329 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.435274    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kzd7\" (UniqueName: \"kubernetes.io/projected/9e187191-f813-4a43-95d8-db851971c2ed-kube-api-access-6kzd7\") pod \"maas-api-revoke-sub-d5215243-8hngd\" (UID: \"9e187191-f813-4a43-95d8-db851971c2ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5215243-8hngd"
Oct 30 19:00:00.435329 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.435318    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e187191-f813-4a43-95d8-db851971c2ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5215243-8hngd\" (UID: \"9e187191-f813-4a43-95d8-db851971c2ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5215243-8hngd"
Oct 30 19:00:00.435486 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.435443    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e187191-f813-4a43-95d8-db851971c2ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5215243-8hngd\" (UID: \"9e187191-f813-4a43-95d8-db851971c2ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5215243-8hngd"
Oct 30 19:00:00.435578 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.435528    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7582ec87-a899-40e7-9fa8-793e9caaf2f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g\" (UID: \"7582ec87-a899-40e7-9fa8-793e9caaf2f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g"
Oct 30 19:00:00.435634 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.435585    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56c52368-2c45-4ef9-aa51-d507aafee931-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd\" (UID: \"56c52368-2c45-4ef9-aa51-d507aafee931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd"
Oct 30 19:00:00.436127 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.436107    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7582ec87-a899-40e7-9fa8-793e9caaf2f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g\" (UID: \"7582ec87-a899-40e7-9fa8-793e9caaf2f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g"
Oct 30 19:00:00.436357 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.436334    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56c52368-2c45-4ef9-aa51-d507aafee931-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd\" (UID: \"56c52368-2c45-4ef9-aa51-d507aafee931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd"
Oct 30 19:00:00.458294 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.458242    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5215243-8hngd"]
Oct 30 19:00:00.464617 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.464596    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:00:00.489737 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.489718    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgxdc\" (UniqueName: \"kubernetes.io/projected/56c52368-2c45-4ef9-aa51-d507aafee931-kube-api-access-rgxdc\") pod \"maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd\" (UID: \"56c52368-2c45-4ef9-aa51-d507aafee931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd"
Oct 30 19:00:00.489823 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.489807    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jxgm\" (UniqueName: \"kubernetes.io/projected/7582ec87-a899-40e7-9fa8-793e9caaf2f2-kube-api-access-2jxgm\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g\" (UID: \"7582ec87-a899-40e7-9fa8-793e9caaf2f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g"
Oct 30 19:00:00.511144 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.511125    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl"
Oct 30 19:00:00.518934 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.518915    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576"
Oct 30 19:00:00.530396 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.528896    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc"
Oct 30 19:00:00.536269 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.536243    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kzd7\" (UniqueName: \"kubernetes.io/projected/9e187191-f813-4a43-95d8-db851971c2ed-kube-api-access-6kzd7\") pod \"maas-api-revoke-sub-d5215243-8hngd\" (UID: \"9e187191-f813-4a43-95d8-db851971c2ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5215243-8hngd"
Oct 30 19:00:00.536359 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.536289    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e187191-f813-4a43-95d8-db851971c2ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5215243-8hngd\" (UID: \"9e187191-f813-4a43-95d8-db851971c2ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5215243-8hngd"
Oct 30 19:00:00.536359 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.536328    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e187191-f813-4a43-95d8-db851971c2ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5215243-8hngd\" (UID: \"9e187191-f813-4a43-95d8-db851971c2ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5215243-8hngd"
Oct 30 19:00:00.537141 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.537117    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e187191-f813-4a43-95d8-db851971c2ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5215243-8hngd\" (UID: \"9e187191-f813-4a43-95d8-db851971c2ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5215243-8hngd"
Oct 30 19:00:00.539010 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.538988    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e187191-f813-4a43-95d8-db851971c2ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5215243-8hngd\" (UID: \"9e187191-f813-4a43-95d8-db851971c2ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5215243-8hngd"
Oct 30 19:00:00.551681 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.551653    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd"
Oct 30 19:00:00.559227 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.559198    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g"
Oct 30 19:00:00.591553 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.591522    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kzd7\" (UniqueName: \"kubernetes.io/projected/9e187191-f813-4a43-95d8-db851971c2ed-kube-api-access-6kzd7\") pod \"maas-api-revoke-sub-d5215243-8hngd\" (UID: \"9e187191-f813-4a43-95d8-db851971c2ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5215243-8hngd"
Oct 30 19:00:00.736747 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.736675    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5215243-8hngd"
Oct 30 19:00:00.931724 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.931692    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl"]
Oct 30 19:00:00.982330 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:00:00.982305    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c52368_2c45_4ef9_aa51_d507aafee931.slice/crio-de8f83d1a2fac28a4e4f746a7c81a10b8da5da54d84cbb4ba26aa513fb4fa61b WatchSource:0}: Error finding container de8f83d1a2fac28a4e4f746a7c81a10b8da5da54d84cbb4ba26aa513fb4fa61b: Status 404 returned error can't find the container with id de8f83d1a2fac28a4e4f746a7c81a10b8da5da54d84cbb4ba26aa513fb4fa61b
Oct 30 19:00:00.983715 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:00:00.983693    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba14910_fb5d_40ec_bad0_39dfebd1506e.slice/crio-6a717b9f4dc09eeecff4507ee78edb3d4a75531fd2ef941e46e22d4f11f7069d WatchSource:0}: Error finding container 6a717b9f4dc09eeecff4507ee78edb3d4a75531fd2ef941e46e22d4f11f7069d: Status 404 returned error can't find the container with id 6a717b9f4dc09eeecff4507ee78edb3d4a75531fd2ef941e46e22d4f11f7069d
Oct 30 19:00:00.983825 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.983730    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576"]
Oct 30 19:00:00.984706 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:00:00.984654    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f224d4_04cc_4493_b313_56626b0f50a1.slice/crio-3a85787b240e2f5ad3e10331186e36bace9443f93035180a6e9a3e15e91966c1 WatchSource:0}: Error finding container 3a85787b240e2f5ad3e10331186e36bace9443f93035180a6e9a3e15e91966c1: Status 404 returned error can't find the container with id 3a85787b240e2f5ad3e10331186e36bace9443f93035180a6e9a3e15e91966c1
Oct 30 19:00:00.990244 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.990224    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc"]
Oct 30 19:00:00.996603 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:00.996579    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd"]
Oct 30 19:00:01.022241 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:01.022210    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g"]
Oct 30 19:00:01.025520 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:00:01.025413    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7582ec87_a899_40e7_9fa8_793e9caaf2f2.slice/crio-4836997fbe8551100a8a335aa060a3d6bc417927705b714c82cb188a8ddb0a62 WatchSource:0}: Error finding container 4836997fbe8551100a8a335aa060a3d6bc417927705b714c82cb188a8ddb0a62: Status 404 returned error can't find the container with id 4836997fbe8551100a8a335aa060a3d6bc417927705b714c82cb188a8ddb0a62
Oct 30 19:00:01.129112 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:01.129032    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21eb612d-jjp9d"]
Oct 30 19:00:01.148514 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:01.148488    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21eb612d-jjp9d"]
Oct 30 19:00:01.152677 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:01.152654    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5215243-8hngd"]
Oct 30 19:00:01.208027 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:00:01.207998    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e187191_f813_4a43_95d8_db851971c2ed.slice/crio-65cf67527cb280761ea9aedb82107cad1c08b31fc36efa72d025e14bb529a5f1 WatchSource:0}: Error finding container 65cf67527cb280761ea9aedb82107cad1c08b31fc36efa72d025e14bb529a5f1: Status 404 returned error can't find the container with id 65cf67527cb280761ea9aedb82107cad1c08b31fc36efa72d025e14bb529a5f1
Oct 30 19:00:01.300507 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:01.300419    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576" event={"ID":"c0f224d4-04cc-4493-b313-56626b0f50a1","Type":"ContainerStarted","Data":"3a85787b240e2f5ad3e10331186e36bace9443f93035180a6e9a3e15e91966c1"}
Oct 30 19:00:01.301719 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:01.301694    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5215243-8hngd" event={"ID":"9e187191-f813-4a43-95d8-db851971c2ed","Type":"ContainerStarted","Data":"65cf67527cb280761ea9aedb82107cad1c08b31fc36efa72d025e14bb529a5f1"}
Oct 30 19:00:01.303372 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:01.303342    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd" event={"ID":"56c52368-2c45-4ef9-aa51-d507aafee931","Type":"ContainerStarted","Data":"de8f83d1a2fac28a4e4f746a7c81a10b8da5da54d84cbb4ba26aa513fb4fa61b"}
Oct 30 19:00:01.304979 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:01.304951    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g" event={"ID":"7582ec87-a899-40e7-9fa8-793e9caaf2f2","Type":"ContainerStarted","Data":"4836997fbe8551100a8a335aa060a3d6bc417927705b714c82cb188a8ddb0a62"}
Oct 30 19:00:01.306476 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:01.306452    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc" event={"ID":"1ba14910-fb5d-40ec-bad0-39dfebd1506e","Type":"ContainerStarted","Data":"6a717b9f4dc09eeecff4507ee78edb3d4a75531fd2ef941e46e22d4f11f7069d"}
Oct 30 19:00:01.307809 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:01.307787    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl" event={"ID":"884e56e6-4b28-4245-a418-1782703a11df","Type":"ContainerStarted","Data":"14045db139cc8b3ce68d3b272c844551f71b18de810e1882607f92e8ccd8730b"}
Oct 30 19:00:01.993303 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:01.993272    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:02.317256 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:02.317216    4924 generic.go:356] "Generic (PLEG): container finished" podID="7582ec87-a899-40e7-9fa8-793e9caaf2f2" containerID="d18ea42d5ff9b3dde204bc0c6051a71de14defdd2eaf0eb2411a61360451b2f2" exitCode=60
Oct 30 19:00:02.317694 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:02.317282    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g" event={"ID":"7582ec87-a899-40e7-9fa8-793e9caaf2f2","Type":"ContainerDied","Data":"d18ea42d5ff9b3dde204bc0c6051a71de14defdd2eaf0eb2411a61360451b2f2"}
Oct 30 19:00:02.317694 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:02.317666    4924 scope.go:122] "RemoveContainer" containerID="d18ea42d5ff9b3dde204bc0c6051a71de14defdd2eaf0eb2411a61360451b2f2"
Oct 30 19:00:02.319022 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:02.319002    4924 generic.go:356] "Generic (PLEG): container finished" podID="1ba14910-fb5d-40ec-bad0-39dfebd1506e" containerID="48fda493ede920e73d2f6d5dc23bd1fd0d3e4ce326a9ed0b43e8d23c15058951" exitCode=60
Oct 30 19:00:02.319163 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:02.319148    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc" event={"ID":"1ba14910-fb5d-40ec-bad0-39dfebd1506e","Type":"ContainerDied","Data":"48fda493ede920e73d2f6d5dc23bd1fd0d3e4ce326a9ed0b43e8d23c15058951"}
Oct 30 19:00:02.319416 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:02.319341    4924 scope.go:122] "RemoveContainer" containerID="48fda493ede920e73d2f6d5dc23bd1fd0d3e4ce326a9ed0b43e8d23c15058951"
Oct 30 19:00:02.320907 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:02.320887    4924 generic.go:356] "Generic (PLEG): container finished" podID="884e56e6-4b28-4245-a418-1782703a11df" containerID="7fabb5b0bc6403a2419e98651ca168d903ec467cf42f6c73f7a4f2d11811630e" exitCode=60
Oct 30 19:00:02.321054 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:02.321013    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl" event={"ID":"884e56e6-4b28-4245-a418-1782703a11df","Type":"ContainerDied","Data":"7fabb5b0bc6403a2419e98651ca168d903ec467cf42f6c73f7a4f2d11811630e"}
Oct 30 19:00:02.321327 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:02.321283    4924 scope.go:122] "RemoveContainer" containerID="7fabb5b0bc6403a2419e98651ca168d903ec467cf42f6c73f7a4f2d11811630e"
Oct 30 19:00:02.323381 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:02.323360    4924 generic.go:356] "Generic (PLEG): container finished" podID="c0f224d4-04cc-4493-b313-56626b0f50a1" containerID="c94f311bea0758d2fcc3d528b9b9630eac34bc02340aeebea55253662454b13b" exitCode=60
Oct 30 19:00:02.323461 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:02.323385    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576" event={"ID":"c0f224d4-04cc-4493-b313-56626b0f50a1","Type":"ContainerDied","Data":"c94f311bea0758d2fcc3d528b9b9630eac34bc02340aeebea55253662454b13b"}
Oct 30 19:00:02.323661 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:02.323641    4924 scope.go:122] "RemoveContainer" containerID="c94f311bea0758d2fcc3d528b9b9630eac34bc02340aeebea55253662454b13b"
Oct 30 19:00:02.325184 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:02.325161    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5215243-8hngd" event={"ID":"9e187191-f813-4a43-95d8-db851971c2ed","Type":"ContainerStarted","Data":"22ea07c211abac5dbf801704eadca516cf07ff14030f54cd706af40931d94c20"}
Oct 30 19:00:02.326678 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:02.326660    4924 generic.go:356] "Generic (PLEG): container finished" podID="56c52368-2c45-4ef9-aa51-d507aafee931" containerID="c14c2789dd607331e1f19292c48d2bd43892955dfb75ceade813e730a5e16c8e" exitCode=60
Oct 30 19:00:02.326781 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:02.326741    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd" event={"ID":"56c52368-2c45-4ef9-aa51-d507aafee931","Type":"ContainerDied","Data":"c14c2789dd607331e1f19292c48d2bd43892955dfb75ceade813e730a5e16c8e"}
Oct 30 19:00:02.326941 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:02.326925    4924 scope.go:122] "RemoveContainer" containerID="c14c2789dd607331e1f19292c48d2bd43892955dfb75ceade813e730a5e16c8e"
Oct 30 19:00:02.845630 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:02.845593    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eb70edf-668c-4255-a633-e52ea0f587da" path="/var/lib/kubelet/pods/7eb70edf-668c-4255-a633-e52ea0f587da/volumes"
Oct 30 19:00:02.997707 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:02.997653    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5215243-8hngd" podStartSLOduration=2.997638562 podStartE2EDuration="2.997638562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:02.994525121 +0000 UTC m=+1404.271575566" watchObservedRunningTime="2026-10-30 19:00:02.997638562 +0000 UTC m=+1404.274689007"
Oct 30 19:00:03.336756 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:03.336728    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl" event={"ID":"884e56e6-4b28-4245-a418-1782703a11df","Type":"ContainerStarted","Data":"8241af4014084cb2bebe86256e514a7dfaad45074f1b0d11009e8f5857d59491"}
Oct 30 19:00:03.339568 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:03.339541    4924 generic.go:356] "Generic (PLEG): container finished" podID="c0f224d4-04cc-4493-b313-56626b0f50a1" containerID="a1d45025556ad2a57c5b153599b50fc04b182cfd021443c687a2b831e0b87d19" exitCode=60
Oct 30 19:00:03.339656 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:03.339616    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576" event={"ID":"c0f224d4-04cc-4493-b313-56626b0f50a1","Type":"ContainerDied","Data":"a1d45025556ad2a57c5b153599b50fc04b182cfd021443c687a2b831e0b87d19"}
Oct 30 19:00:03.339712 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:03.339669    4924 scope.go:122] "RemoveContainer" containerID="c94f311bea0758d2fcc3d528b9b9630eac34bc02340aeebea55253662454b13b"
Oct 30 19:00:03.339953 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:03.339934    4924 scope.go:122] "RemoveContainer" containerID="a1d45025556ad2a57c5b153599b50fc04b182cfd021443c687a2b831e0b87d19"
Oct 30 19:00:03.340348 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:00:03.340322    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576_odh-ai-gateway-infra(c0f224d4-04cc-4493-b313-56626b0f50a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576" podUID="c0f224d4-04cc-4493-b313-56626b0f50a1"
Oct 30 19:00:03.341297 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:03.341273    4924 generic.go:356] "Generic (PLEG): container finished" podID="9e187191-f813-4a43-95d8-db851971c2ed" containerID="22ea07c211abac5dbf801704eadca516cf07ff14030f54cd706af40931d94c20" exitCode=0
Oct 30 19:00:03.341381 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:03.341319    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5215243-8hngd" event={"ID":"9e187191-f813-4a43-95d8-db851971c2ed","Type":"ContainerDied","Data":"22ea07c211abac5dbf801704eadca516cf07ff14030f54cd706af40931d94c20"}
Oct 30 19:00:03.343080 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:03.343047    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd" event={"ID":"56c52368-2c45-4ef9-aa51-d507aafee931","Type":"ContainerStarted","Data":"54ba0e05e84fad67b4ce6977ad4e521d903c9545518a180fb34349d85dcbb1de"}
Oct 30 19:00:03.345059 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:03.345033    4924 generic.go:356] "Generic (PLEG): container finished" podID="7582ec87-a899-40e7-9fa8-793e9caaf2f2" containerID="3772ebcc7753eb75d9de27de150abbcd7810ae3a9146d2775870b4db17871df0" exitCode=60
Oct 30 19:00:03.345175 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:03.345153    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g" event={"ID":"7582ec87-a899-40e7-9fa8-793e9caaf2f2","Type":"ContainerDied","Data":"3772ebcc7753eb75d9de27de150abbcd7810ae3a9146d2775870b4db17871df0"}
Oct 30 19:00:03.345400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:03.345385    4924 scope.go:122] "RemoveContainer" containerID="3772ebcc7753eb75d9de27de150abbcd7810ae3a9146d2775870b4db17871df0"
Oct 30 19:00:03.345695 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:00:03.345672    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g_odh-ai-gateway-infra(7582ec87-a899-40e7-9fa8-793e9caaf2f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g" podUID="7582ec87-a899-40e7-9fa8-793e9caaf2f2"
Oct 30 19:00:03.347054 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:03.347034    4924 generic.go:356] "Generic (PLEG): container finished" podID="1ba14910-fb5d-40ec-bad0-39dfebd1506e" containerID="b85170ed608aaa3dba3e43804c4b6e301f53f24446bd535fb72fc0d2bc74b61d" exitCode=60
Oct 30 19:00:03.347161 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:03.347105    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc" event={"ID":"1ba14910-fb5d-40ec-bad0-39dfebd1506e","Type":"ContainerDied","Data":"b85170ed608aaa3dba3e43804c4b6e301f53f24446bd535fb72fc0d2bc74b61d"}
Oct 30 19:00:03.347343 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:03.347329    4924 scope.go:122] "RemoveContainer" containerID="b85170ed608aaa3dba3e43804c4b6e301f53f24446bd535fb72fc0d2bc74b61d"
Oct 30 19:00:03.347569 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:00:03.347553    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc_odh-ai-gateway-infra(1ba14910-fb5d-40ec-bad0-39dfebd1506e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc" podUID="1ba14910-fb5d-40ec-bad0-39dfebd1506e"
Oct 30 19:00:03.371899 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:03.371870    4924 scope.go:122] "RemoveContainer" containerID="d18ea42d5ff9b3dde204bc0c6051a71de14defdd2eaf0eb2411a61360451b2f2"
Oct 30 19:00:03.387998 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:03.387979    4924 scope.go:122] "RemoveContainer" containerID="48fda493ede920e73d2f6d5dc23bd1fd0d3e4ce326a9ed0b43e8d23c15058951"
Oct 30 19:00:03.449302 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:03.449253    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl" podStartSLOduration=3.449239608 podStartE2EDuration="3.449239608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:03.44454011 +0000 UTC m=+1404.721590555" watchObservedRunningTime="2026-10-30 19:00:03.449239608 +0000 UTC m=+1404.726290052"
Oct 30 19:00:03.775557 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:03.775461    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd" podStartSLOduration=3.775446134 podStartE2EDuration="3.775446134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:03.772088956 +0000 UTC m=+1405.049139402" watchObservedRunningTime="2026-10-30 19:00:03.775446134 +0000 UTC m=+1405.052496579"
Oct 30 19:00:04.357734 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:04.357702    4924 scope.go:122] "RemoveContainer" containerID="b85170ed608aaa3dba3e43804c4b6e301f53f24446bd535fb72fc0d2bc74b61d"
Oct 30 19:00:04.358211 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:00:04.358152    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc_odh-ai-gateway-infra(1ba14910-fb5d-40ec-bad0-39dfebd1506e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc" podUID="1ba14910-fb5d-40ec-bad0-39dfebd1506e"
Oct 30 19:00:04.359505 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:04.359482    4924 generic.go:356] "Generic (PLEG): container finished" podID="884e56e6-4b28-4245-a418-1782703a11df" containerID="8241af4014084cb2bebe86256e514a7dfaad45074f1b0d11009e8f5857d59491" exitCode=60
Oct 30 19:00:04.359637 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:04.359566    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl" event={"ID":"884e56e6-4b28-4245-a418-1782703a11df","Type":"ContainerDied","Data":"8241af4014084cb2bebe86256e514a7dfaad45074f1b0d11009e8f5857d59491"}
Oct 30 19:00:04.359637 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:04.359629    4924 scope.go:122] "RemoveContainer" containerID="7fabb5b0bc6403a2419e98651ca168d903ec467cf42f6c73f7a4f2d11811630e"
Oct 30 19:00:04.359854 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:04.359837    4924 scope.go:122] "RemoveContainer" containerID="8241af4014084cb2bebe86256e514a7dfaad45074f1b0d11009e8f5857d59491"
Oct 30 19:00:04.360190 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:00:04.360165    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl_odh-ai-gateway-infra(884e56e6-4b28-4245-a418-1782703a11df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl" podUID="884e56e6-4b28-4245-a418-1782703a11df"
Oct 30 19:00:04.362026 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:04.361997    4924 scope.go:122] "RemoveContainer" containerID="a1d45025556ad2a57c5b153599b50fc04b182cfd021443c687a2b831e0b87d19"
Oct 30 19:00:04.362399 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:00:04.362373    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576_odh-ai-gateway-infra(c0f224d4-04cc-4493-b313-56626b0f50a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576" podUID="c0f224d4-04cc-4493-b313-56626b0f50a1"
Oct 30 19:00:04.363837 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:04.363819    4924 generic.go:356] "Generic (PLEG): container finished" podID="56c52368-2c45-4ef9-aa51-d507aafee931" containerID="54ba0e05e84fad67b4ce6977ad4e521d903c9545518a180fb34349d85dcbb1de" exitCode=60
Oct 30 19:00:04.363921 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:04.363902    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd" event={"ID":"56c52368-2c45-4ef9-aa51-d507aafee931","Type":"ContainerDied","Data":"54ba0e05e84fad67b4ce6977ad4e521d903c9545518a180fb34349d85dcbb1de"}
Oct 30 19:00:04.364121 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:04.364101    4924 scope.go:122] "RemoveContainer" containerID="54ba0e05e84fad67b4ce6977ad4e521d903c9545518a180fb34349d85dcbb1de"
Oct 30 19:00:04.364399 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:00:04.364379    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd_odh-ai-gateway-infra(56c52368-2c45-4ef9-aa51-d507aafee931)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd" podUID="56c52368-2c45-4ef9-aa51-d507aafee931"
Oct 30 19:00:04.365962 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:04.365933    4924 scope.go:122] "RemoveContainer" containerID="3772ebcc7753eb75d9de27de150abbcd7810ae3a9146d2775870b4db17871df0"
Oct 30 19:00:04.366214 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:00:04.366154    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g_odh-ai-gateway-infra(7582ec87-a899-40e7-9fa8-793e9caaf2f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g" podUID="7582ec87-a899-40e7-9fa8-793e9caaf2f2"
Oct 30 19:00:04.374177 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:04.374158    4924 scope.go:122] "RemoveContainer" containerID="c14c2789dd607331e1f19292c48d2bd43892955dfb75ceade813e730a5e16c8e"
Oct 30 19:00:04.529449 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:04.529414    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5215243-8hngd"
Oct 30 19:00:04.594467 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:04.594424    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e187191-f813-4a43-95d8-db851971c2ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e187191-f813-4a43-95d8-db851971c2ed-maas-api-openshift-ca\") pod \"9e187191-f813-4a43-95d8-db851971c2ed\" (UID: \"9e187191-f813-4a43-95d8-db851971c2ed\") "
Oct 30 19:00:04.594467 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:04.594478    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e187191-f813-4a43-95d8-db851971c2ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e187191-f813-4a43-95d8-db851971c2ed-maas-api-xks-ca\") pod \"9e187191-f813-4a43-95d8-db851971c2ed\" (UID: \"9e187191-f813-4a43-95d8-db851971c2ed\") "
Oct 30 19:00:04.594723 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:04.594537    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e187191-f813-4a43-95d8-db851971c2ed-kube-api-access-6kzd7\" (UniqueName: \"kubernetes.io/projected/9e187191-f813-4a43-95d8-db851971c2ed-kube-api-access-6kzd7\") pod \"9e187191-f813-4a43-95d8-db851971c2ed\" (UID: \"9e187191-f813-4a43-95d8-db851971c2ed\") "
Oct 30 19:00:04.594866 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:04.594839    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e187191-f813-4a43-95d8-db851971c2ed-maas-api-openshift-ca" pod "9e187191-f813-4a43-95d8-db851971c2ed" (UID: "9e187191-f813-4a43-95d8-db851971c2ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:04.595102 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:04.595083    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e187191-f813-4a43-95d8-db851971c2ed-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:04.596644 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:04.596625    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e187191-f813-4a43-95d8-db851971c2ed-maas-api-xks-ca" pod "9e187191-f813-4a43-95d8-db851971c2ed" (UID: "9e187191-f813-4a43-95d8-db851971c2ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:04.596704 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:04.596652    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e187191-f813-4a43-95d8-db851971c2ed-kube-api-access-6kzd7" pod "9e187191-f813-4a43-95d8-db851971c2ed" (UID: "9e187191-f813-4a43-95d8-db851971c2ed"). InnerVolumeSpecName "kube-api-access-6kzd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:04.702606 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:04.702514    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e187191-f813-4a43-95d8-db851971c2ed-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:04.702606 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:04.702548    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kzd7\" (UniqueName: \"kubernetes.io/projected/9e187191-f813-4a43-95d8-db851971c2ed-kube-api-access-6kzd7\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:05.376036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:05.376006    4924 scope.go:122] "RemoveContainer" containerID="8241af4014084cb2bebe86256e514a7dfaad45074f1b0d11009e8f5857d59491"
Oct 30 19:00:05.376485 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:00:05.376396    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl_odh-ai-gateway-infra(884e56e6-4b28-4245-a418-1782703a11df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl" podUID="884e56e6-4b28-4245-a418-1782703a11df"
Oct 30 19:00:05.377283 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:05.377252    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5215243-8hngd" event={"ID":"9e187191-f813-4a43-95d8-db851971c2ed","Type":"ContainerDied","Data":"65cf67527cb280761ea9aedb82107cad1c08b31fc36efa72d025e14bb529a5f1"}
Oct 30 19:00:05.377413 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:05.377307    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65cf67527cb280761ea9aedb82107cad1c08b31fc36efa72d025e14bb529a5f1"
Oct 30 19:00:05.377413 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:05.377263    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5215243-8hngd"
Oct 30 19:00:05.379298 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:05.379272    4924 scope.go:122] "RemoveContainer" containerID="54ba0e05e84fad67b4ce6977ad4e521d903c9545518a180fb34349d85dcbb1de"
Oct 30 19:00:05.379583 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:00:05.379554    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd_odh-ai-gateway-infra(56c52368-2c45-4ef9-aa51-d507aafee931)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd" podUID="56c52368-2c45-4ef9-aa51-d507aafee931"
Oct 30 19:00:06.586925 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:06.581560    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:06.909749 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:06.909711    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:07.641639 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:07.641594    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfe70bae-bqfq2"]
Oct 30 19:00:07.643118 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:07.643090    4924 cpu_manager.go:414] "containerMap: removing container" podUID="9e187191-f813-4a43-95d8-db851971c2ed" containerName="revoke-keys"
Oct 30 19:00:07.643118 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:07.643119    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e187191-f813-4a43-95d8-db851971c2ed" containerName="revoke-keys"
Oct 30 19:00:07.643509 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:07.643490    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e187191-f813-4a43-95d8-db851971c2ed" containerName="revoke-keys"
Oct 30 19:00:07.649115 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:07.649091    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe70bae-bqfq2"
Oct 30 19:00:07.695579 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:07.695542    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfe70bae-bqfq2"]
Oct 30 19:00:07.736184 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:07.736145    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:00:07.748343 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:07.748314    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f97819-2d12-424c-b617-936dd6f9099b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfe70bae-bqfq2\" (UID: \"27f97819-2d12-424c-b617-936dd6f9099b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe70bae-bqfq2"
Oct 30 19:00:07.748541 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:07.748354    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vh78\" (UniqueName: \"kubernetes.io/projected/27f97819-2d12-424c-b617-936dd6f9099b-kube-api-access-2vh78\") pod \"maas-api-revoke-sub-dfe70bae-bqfq2\" (UID: \"27f97819-2d12-424c-b617-936dd6f9099b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe70bae-bqfq2"
Oct 30 19:00:07.748618 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:07.748540    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f97819-2d12-424c-b617-936dd6f9099b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfe70bae-bqfq2\" (UID: \"27f97819-2d12-424c-b617-936dd6f9099b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe70bae-bqfq2"
Oct 30 19:00:07.850552 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:07.850518    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f97819-2d12-424c-b617-936dd6f9099b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfe70bae-bqfq2\" (UID: \"27f97819-2d12-424c-b617-936dd6f9099b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe70bae-bqfq2"
Oct 30 19:00:07.850748 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:07.850731    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f97819-2d12-424c-b617-936dd6f9099b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfe70bae-bqfq2\" (UID: \"27f97819-2d12-424c-b617-936dd6f9099b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe70bae-bqfq2"
Oct 30 19:00:07.850838 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:07.850775    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vh78\" (UniqueName: \"kubernetes.io/projected/27f97819-2d12-424c-b617-936dd6f9099b-kube-api-access-2vh78\") pod \"maas-api-revoke-sub-dfe70bae-bqfq2\" (UID: \"27f97819-2d12-424c-b617-936dd6f9099b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe70bae-bqfq2"
Oct 30 19:00:07.851353 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:07.851329    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f97819-2d12-424c-b617-936dd6f9099b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfe70bae-bqfq2\" (UID: \"27f97819-2d12-424c-b617-936dd6f9099b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe70bae-bqfq2"
Oct 30 19:00:07.853262 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:07.853239    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f97819-2d12-424c-b617-936dd6f9099b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfe70bae-bqfq2\" (UID: \"27f97819-2d12-424c-b617-936dd6f9099b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe70bae-bqfq2"
Oct 30 19:00:07.905363 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:07.905294    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vh78\" (UniqueName: \"kubernetes.io/projected/27f97819-2d12-424c-b617-936dd6f9099b-kube-api-access-2vh78\") pod \"maas-api-revoke-sub-dfe70bae-bqfq2\" (UID: \"27f97819-2d12-424c-b617-936dd6f9099b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe70bae-bqfq2"
Oct 30 19:00:07.963233 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:07.963201    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe70bae-bqfq2"
Oct 30 19:00:08.339611 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:00:08.339582    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f97819_2d12_424c_b617_936dd6f9099b.slice/crio-1d960451d92b2350eab5a748865dab2e675b3f6e45bf33339a83f4d5187abf21 WatchSource:0}: Error finding container 1d960451d92b2350eab5a748865dab2e675b3f6e45bf33339a83f4d5187abf21: Status 404 returned error can't find the container with id 1d960451d92b2350eab5a748865dab2e675b3f6e45bf33339a83f4d5187abf21
Oct 30 19:00:08.340477 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:08.340385    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfe70bae-bqfq2"]
Oct 30 19:00:08.404911 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:08.404880    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe70bae-bqfq2" event={"ID":"27f97819-2d12-424c-b617-936dd6f9099b","Type":"ContainerStarted","Data":"1d960451d92b2350eab5a748865dab2e675b3f6e45bf33339a83f4d5187abf21"}
Oct 30 19:00:09.054284 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:09.054198    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac2a0f50-xw5xh"]
Oct 30 19:00:09.102867 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:09.102836    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a7b9ed6-8tvfd"]
Oct 30 19:00:09.109389 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:09.109355    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac2a0f50-xw5xh"]
Oct 30 19:00:09.113453 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:09.113422    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a7b9ed6-8tvfd"]
Oct 30 19:00:09.414974 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:09.414924    4924 generic.go:356] "Generic (PLEG): container finished" podID="27f97819-2d12-424c-b617-936dd6f9099b" containerID="dbf7cefba4e8666eef1fb7c3d45d8896e9f4b69d159ba1ac9b2854eab135c002" exitCode=0
Oct 30 19:00:09.415182 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:09.415015    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe70bae-bqfq2" event={"ID":"27f97819-2d12-424c-b617-936dd6f9099b","Type":"ContainerDied","Data":"dbf7cefba4e8666eef1fb7c3d45d8896e9f4b69d159ba1ac9b2854eab135c002"}
Oct 30 19:00:10.292775 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.292737    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"]
Oct 30 19:00:10.297332 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.297311    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.339328 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.339304    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"]
Oct 30 19:00:10.345255 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.345238    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.383979 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.383947    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-istio-data\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.384158 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.383997    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-credential-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.384158 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.384043    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-workload-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.384158 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.384156    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-workload-certs\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.384347 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.384182    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53b2edae-97f1-44e9-9580-29d605357c8f-istio-podinfo\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.384347 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.384275    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-istio-envoy\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.384444 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.384350    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53b2edae-97f1-44e9-9580-29d605357c8f-istio-token\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.384444 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.384365    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53b2edae-97f1-44e9-9580-29d605357c8f-istiod-ca-cert\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.384444 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.384397    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vwbp\" (UniqueName: \"kubernetes.io/projected/53b2edae-97f1-44e9-9580-29d605357c8f-kube-api-access-5vwbp\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.409528 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.397834    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"]
Oct 30 19:00:10.412335 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.412312    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.443175 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.443144    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"]
Oct 30 19:00:10.511215 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.511178    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53b2edae-97f1-44e9-9580-29d605357c8f-istio-token\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.511508 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.511478    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53b2edae-97f1-44e9-9580-29d605357c8f-istiod-ca-cert\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.511691 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.511658    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vwbp\" (UniqueName: \"kubernetes.io/projected/53b2edae-97f1-44e9-9580-29d605357c8f-kube-api-access-5vwbp\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.513026 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.513004    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-workload-certs\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.513283 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.513262    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-workload-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.513459 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.513444    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-istio-data\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.513604 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.513578    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-token\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.513713 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.513699    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7gqp\" (UniqueName: \"kubernetes.io/projected/51837408-10f0-4026-b8c6-fcb002a1e459-kube-api-access-n7gqp\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.513820 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.513806    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-credential-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.513949 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.513934    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-workload-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.514094 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.514076    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-workload-certs\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.514242 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.514228    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51837408-10f0-4026-b8c6-fcb002a1e459-istiod-ca-cert\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.514354 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.514341    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-istio-envoy\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.514487 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.514473    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-data\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.517622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.517589    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-workload-certs\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.517774 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.517759    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53b2edae-97f1-44e9-9580-29d605357c8f-istio-podinfo\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.517910 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.517895    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-envoy\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.518032 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.518016    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-istio-data\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.518163 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.518146    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922dfc11-a4ba-4d50-a50f-b5155d277792-istiod-ca-cert\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.518287 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.518273    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mchj\" (UniqueName: \"kubernetes.io/projected/922dfc11-a4ba-4d50-a50f-b5155d277792-kube-api-access-9mchj\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.518426 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.518395    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-credential-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.518550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.518534    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-credential-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.518762 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.518685    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51837408-10f0-4026-b8c6-fcb002a1e459-istio-podinfo\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.518933 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.518917    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51837408-10f0-4026-b8c6-fcb002a1e459-istio-token\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.519904 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.519884    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-istio-envoy\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.520431 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.520401    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-podinfo\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.520673 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.520653    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-workload-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.521008 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.515054    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-istio-data\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.521155 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.515360    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-workload-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.521274 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.512740    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53b2edae-97f1-44e9-9580-29d605357c8f-istiod-ca-cert\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.521384 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.514845    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-credential-socket\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.521536 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.519321    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-workload-certs\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.564458 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.564286    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53b2edae-97f1-44e9-9580-29d605357c8f-istio-podinfo\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.565370 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.565331    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-istio-envoy\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.573106 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.567769    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vwbp\" (UniqueName: \"kubernetes.io/projected/53b2edae-97f1-44e9-9580-29d605357c8f-kube-api-access-5vwbp\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.596086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.586232    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"]
Oct 30 19:00:10.601135 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.601105    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"]
Oct 30 19:00:10.601612 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.601591    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53b2edae-97f1-44e9-9580-29d605357c8f-istio-token\") pod \"e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") " pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.620130 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.616418    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:10.643338 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.643282    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-workload-certs\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.643652 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.643584    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51837408-10f0-4026-b8c6-fcb002a1e459-istiod-ca-cert\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.644034 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.643813    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-istio-envoy\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.645092 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.644195    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-data\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.645092 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.644315    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-envoy\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.645092 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.644346    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-istio-data\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.645092 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.644377    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922dfc11-a4ba-4d50-a50f-b5155d277792-istiod-ca-cert\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.645092 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.644414    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mchj\" (UniqueName: \"kubernetes.io/projected/922dfc11-a4ba-4d50-a50f-b5155d277792-kube-api-access-9mchj\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.645092 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.644456    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-credential-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.645092 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.644483    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-credential-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.645092 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.644538    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51837408-10f0-4026-b8c6-fcb002a1e459-istio-podinfo\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.645092 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.644593    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51837408-10f0-4026-b8c6-fcb002a1e459-istio-token\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.645092 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.644683    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-podinfo\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.645092 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.644720    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51837408-10f0-4026-b8c6-fcb002a1e459-istiod-ca-cert\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.645092 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.645019    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-workload-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.645852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.645275    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-workload-certs\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.645852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.644722    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-workload-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.645852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.645398    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-workload-certs\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.645852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.645452    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-workload-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.645852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.645499    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-token\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.645852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.645526    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7gqp\" (UniqueName: \"kubernetes.io/projected/51837408-10f0-4026-b8c6-fcb002a1e459-kube-api-access-n7gqp\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.646194 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.646151    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-workload-certs\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.648332 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.646372    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-workload-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.648332 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.647144    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-data\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.654126 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.653159    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-credential-socket\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.654126 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.653414    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-credential-socket\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.654126 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.654053    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922dfc11-a4ba-4d50-a50f-b5155d277792-istiod-ca-cert\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.658086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.654898    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-istio-data\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.658086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.657809    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-podinfo\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.662087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.659529    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51837408-10f0-4026-b8c6-fcb002a1e459-istio-podinfo\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.662087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.660172    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-envoy\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.665389 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.665363    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-istio-envoy\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.706496 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.706011    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mchj\" (UniqueName: \"kubernetes.io/projected/922dfc11-a4ba-4d50-a50f-b5155d277792-kube-api-access-9mchj\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.707530 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.707461    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-token\") pod \"e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") " pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.710034 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.709354    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7gqp\" (UniqueName: \"kubernetes.io/projected/51837408-10f0-4026-b8c6-fcb002a1e459-kube-api-access-n7gqp\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.710034 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.709955    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51837408-10f0-4026-b8c6-fcb002a1e459-istio-token\") pod \"e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") " pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:10.772106 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.766896    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:10.940742 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.940706    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a1e6fa3-8aac-4733-8e3a-a1417428e1cb" path="/var/lib/kubelet/pods/2a1e6fa3-8aac-4733-8e3a-a1417428e1cb/volumes"
Oct 30 19:00:10.941843 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.941818    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca24639a-0332-4c5d-94a8-c7d8b15c6602" path="/var/lib/kubelet/pods/ca24639a-0332-4c5d-94a8-c7d8b15c6602/volumes"
Oct 30 19:00:10.957732 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.957706    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe70bae-bqfq2"
Oct 30 19:00:10.972900 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.971672    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27f97819-2d12-424c-b617-936dd6f9099b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f97819-2d12-424c-b617-936dd6f9099b-maas-api-xks-ca\") pod \"27f97819-2d12-424c-b617-936dd6f9099b\" (UID: \"27f97819-2d12-424c-b617-936dd6f9099b\") "
Oct 30 19:00:10.972900 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.971851    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f97819-2d12-424c-b617-936dd6f9099b-kube-api-access-2vh78\" (UniqueName: \"kubernetes.io/projected/27f97819-2d12-424c-b617-936dd6f9099b-kube-api-access-2vh78\") pod \"27f97819-2d12-424c-b617-936dd6f9099b\" (UID: \"27f97819-2d12-424c-b617-936dd6f9099b\") "
Oct 30 19:00:10.972900 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.971967    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27f97819-2d12-424c-b617-936dd6f9099b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f97819-2d12-424c-b617-936dd6f9099b-maas-api-openshift-ca\") pod \"27f97819-2d12-424c-b617-936dd6f9099b\" (UID: \"27f97819-2d12-424c-b617-936dd6f9099b\") "
Oct 30 19:00:10.974000 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.973946    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f97819-2d12-424c-b617-936dd6f9099b-maas-api-openshift-ca" pod "27f97819-2d12-424c-b617-936dd6f9099b" (UID: "27f97819-2d12-424c-b617-936dd6f9099b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:10.980365 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.980323    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f97819-2d12-424c-b617-936dd6f9099b-maas-api-xks-ca" pod "27f97819-2d12-424c-b617-936dd6f9099b" (UID: "27f97819-2d12-424c-b617-936dd6f9099b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:10.982596 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:10.982302    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:11.003905 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:11.003848    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f97819-2d12-424c-b617-936dd6f9099b-kube-api-access-2vh78" pod "27f97819-2d12-424c-b617-936dd6f9099b" (UID: "27f97819-2d12-424c-b617-936dd6f9099b"). InnerVolumeSpecName "kube-api-access-2vh78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:11.081217 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:11.077035    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vh78\" (UniqueName: \"kubernetes.io/projected/27f97819-2d12-424c-b617-936dd6f9099b-kube-api-access-2vh78\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:11.081217 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:11.077083    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27f97819-2d12-424c-b617-936dd6f9099b-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:11.081217 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:11.077102    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27f97819-2d12-424c-b617-936dd6f9099b-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:11.434089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:11.432131    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"]
Oct 30 19:00:11.546574 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:11.545768    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:00:11.546574 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:11.545875    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:00:11.546574 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:11.545915    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:00:11.584317 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:11.584266    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx" event={"ID":"53b2edae-97f1-44e9-9580-29d605357c8f","Type":"ContainerStarted","Data":"1f30d3d9714a9cbb4127955dcfea4f0d74b430d9afdcd44f2c8df3f529dc5c0d"}
Oct 30 19:00:11.584537 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:11.584506    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"]
Oct 30 19:00:11.585889 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:11.585860    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:00:11.586122 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:11.586103    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:00:11.595527 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:11.588717    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:00:11.595527 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:11.590295    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe70bae-bqfq2" event={"ID":"27f97819-2d12-424c-b617-936dd6f9099b","Type":"ContainerDied","Data":"1d960451d92b2350eab5a748865dab2e675b3f6e45bf33339a83f4d5187abf21"}
Oct 30 19:00:11.595527 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:11.590341    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d960451d92b2350eab5a748865dab2e675b3f6e45bf33339a83f4d5187abf21"
Oct 30 19:00:11.595527 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:11.590405    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfe70bae-bqfq2"
Oct 30 19:00:11.595527 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:11.591995    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"]
Oct 30 19:00:11.848430 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:11.848385    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:00:11.848705 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:11.848679    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:00:11.848863 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:11.848842    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:00:12.001006 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:12.000940    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:12.673797 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:12.672520    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm" event={"ID":"51837408-10f0-4026-b8c6-fcb002a1e459","Type":"ContainerStarted","Data":"8d0466ae76ba1a576c1de2ecfa36a22c15f4f615c906fcacc789117b67fd2981"}
Oct 30 19:00:12.673797 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:12.672594    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm" event={"ID":"51837408-10f0-4026-b8c6-fcb002a1e459","Type":"ContainerStarted","Data":"0c0d5d3e741c9e90dcac8f6a35b90ce080adac9e95abd83df7c6749b36d2c263"}
Oct 30 19:00:12.691513 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:12.691473    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7" podUID="97a3b018-9083-4b7c-af88-6edddc585d67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:12.706051 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:12.706010    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw" event={"ID":"922dfc11-a4ba-4d50-a50f-b5155d277792","Type":"ContainerStarted","Data":"51132c3f0c9a06d65bd4696f29e0ad1db4388b8946da5d03eb28807462cef97c"}
Oct 30 19:00:12.706343 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:12.706321    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw" event={"ID":"922dfc11-a4ba-4d50-a50f-b5155d277792","Type":"ContainerStarted","Data":"0e92a32e11eb36b4dd02bcf8a9277b0111623f1dfe4c21dd8fc9682d4938d8cf"}
Oct 30 19:00:12.770227 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:12.770178    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd" podUID="03ac954a-d07b-4523-8d5a-a13dffd0bd2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded"
Oct 30 19:00:12.770862 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:12.770839    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:13.078253 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:13.078147    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l" podUID="bdc8df4b-8c94-40b8-b4be-80d1432d87de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Oct 30 19:00:13.078876 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:13.078845    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj" podUID="51f3e639-9cd4-4aaa-a568-f19b6d2704fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 19:00:13.079341 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:12.853248    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld" podUID="f04af09f-b8d2-4b47-b0b3-88eb12adff93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:13.088808 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:12.853287    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" podUID="eb82dd7d-e682-42a2-924e-a84f21172fc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 30 19:00:13.160157 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:12.833139    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd" podUID="03ac954a-d07b-4523-8d5a-a13dffd0bd2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:13.160157 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:13.081828    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld" podUID="f04af09f-b8d2-4b47-b0b3-88eb12adff93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:13.160157 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:13.116376    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd" podUID="a3d792df-177d-4f7a-bba8-f6351175462e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded"
Oct 30 19:00:13.189186 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:13.189046    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m" podUID="1a90412d-c047-422e-b3c7-9546d286c303" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:13.189186 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:13.189175    4924 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 19:00:13.189461 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:13.189218    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm" podUID="51837408-10f0-4026-b8c6-fcb002a1e459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 19:00:13.239090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:13.217630    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2" podUID="fdf4711c-418a-436b-848a-d46974566573" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:13.239090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:13.222125    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm" podStartSLOduration=3.222102355 podStartE2EDuration="3.222102355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:12.826596255 +0000 UTC m=+1414.103646702" watchObservedRunningTime="2026-10-30 19:00:13.222102355 +0000 UTC m=+1414.499152807"
Oct 30 19:00:13.239090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:13.229892    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw" podStartSLOduration=3.229872846 podStartE2EDuration="3.229872846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:13.16574685 +0000 UTC m=+1414.442797297" watchObservedRunningTime="2026-10-30 19:00:13.229872846 +0000 UTC m=+1414.506923291"
Oct 30 19:00:13.372138 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:13.335210    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:13.372422 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:13.372390    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx" event={"ID":"53b2edae-97f1-44e9-9580-29d605357c8f","Type":"ContainerStarted","Data":"5ba0865c313b090504e1e17e13cfe38c45726c245351ba777b4d11feb53c5cb8"}
Oct 30 19:00:13.372576 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:13.370718    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx" podStartSLOduration=3.370696112 podStartE2EDuration="3.370696112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:13.302734612 +0000 UTC m=+1414.579785052" watchObservedRunningTime="2026-10-30 19:00:13.370696112 +0000 UTC m=+1414.647746557"
Oct 30 19:00:13.527196 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:13.527148    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86" podUID="60a58597-fdc4-4722-8d2a-29e475f84a76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:13.527488 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:13.527149    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5" podUID="925a4de9-b451-4263-9a1c-eebbbae4a08c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:13.562089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:13.558346    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7" podUID="97a3b018-9083-4b7c-af88-6edddc585d67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:13.647144 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:13.643156    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:13.700202 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:13.700155    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7" podUID="97a3b018-9083-4b7c-af88-6edddc585d67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:00:13.726088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:13.701125    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 19:00:13.902177 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:13.856214    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt" podUID="955e98b9-864b-4636-9511-8b69f981eaa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:14.097088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.095758    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg" podUID="9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:14.149090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.106160    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd" podUID="a3d792df-177d-4f7a-bba8-f6351175462e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": context deadline exceeded"
Oct 30 19:00:14.149090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.106317    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj" podUID="51f3e639-9cd4-4aaa-a568-f19b6d2704fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:14.149090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.106610    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l" podUID="bdc8df4b-8c94-40b8-b4be-80d1432d87de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Oct 30 19:00:14.149090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.110195    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 19:00:14.149090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.110278    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 19:00:14.149090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.140393    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" podUID="eb82dd7d-e682-42a2-924e-a84f21172fc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:14.149090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.140518    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd" podUID="a3d792df-177d-4f7a-bba8-f6351175462e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded"
Oct 30 19:00:14.205525 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.196899    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m" podUID="1a90412d-c047-422e-b3c7-9546d286c303" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:00:14.257160 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.217183    4924 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 19:00:14.257160 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.217241    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm" podUID="51837408-10f0-4026-b8c6-fcb002a1e459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 19:00:14.353281 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.341016    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 19:00:14.353281 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.341097    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw" podUID="922dfc11-a4ba-4d50-a50f-b5155d277792" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 19:00:14.353281 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.341729    4924 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 19:00:14.353281 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.341769    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx" podUID="53b2edae-97f1-44e9-9580-29d605357c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 19:00:14.358103 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.354196    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2" podUID="fdf4711c-418a-436b-848a-d46974566573" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:14.358103 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.355558    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 19:00:14.358103 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.355622    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 19:00:14.358103 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.355662    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 19:00:14.358103 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.355696    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 19:00:14.380182 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.380138    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5" podUID="925a4de9-b451-4263-9a1c-eebbbae4a08c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:14.493149 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.448234    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt" podUID="955e98b9-864b-4636-9511-8b69f981eaa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:14.575004 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.574953    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86" podUID="60a58597-fdc4-4722-8d2a-29e475f84a76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:14.584748 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.584721    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 19:00:14.639086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.639029    4924 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 19:00:14.639376 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.639328    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx" podUID="53b2edae-97f1-44e9-9580-29d605357c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 19:00:14.694089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.684783    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5" podUID="925a4de9-b451-4263-9a1c-eebbbae4a08c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:14.696882 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.695922    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 19:00:14.709364 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.709317    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7" podUID="97a3b018-9083-4b7c-af88-6edddc585d67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:14.774978 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.774887    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 19:00:14.787183 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.787140    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw" podUID="922dfc11-a4ba-4d50-a50f-b5155d277792" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 19:00:15.082235 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.806085    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd" podUID="03ac954a-d07b-4523-8d5a-a13dffd0bd2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:15.100097 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.890210    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld" podUID="f04af09f-b8d2-4b47-b0b3-88eb12adff93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:00:15.188092 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:14.916585    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt" podUID="955e98b9-864b-4636-9511-8b69f981eaa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 19:00:15.189655 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:15.025640    4924 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 19:00:15.215090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:15.189873    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm" podUID="51837408-10f0-4026-b8c6-fcb002a1e459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 19:00:15.216220 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:15.110702    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l" podUID="bdc8df4b-8c94-40b8-b4be-80d1432d87de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:15.229138 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:15.110739    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj" podUID="51f3e639-9cd4-4aaa-a568-f19b6d2704fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:15.229689 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:15.113474    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg" podUID="9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:15.230006 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:15.083382    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zmwpj" podUID="e7a6031a-754e-44d2-8c29-f882a7fa8c92" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.25:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:15.355286 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:15.355240    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2" podUID="fdf4711c-418a-436b-848a-d46974566573" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:15.380282 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:15.358149    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m" podUID="1a90412d-c047-422e-b3c7-9546d286c303" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:15.381040 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:15.358186    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd" podUID="a3d792df-177d-4f7a-bba8-f6351175462e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded"
Oct 30 19:00:15.381377 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:15.358214    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" podUID="eb82dd7d-e682-42a2-924e-a84f21172fc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 30 19:00:15.610993 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:15.610891    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86" podUID="60a58597-fdc4-4722-8d2a-29e475f84a76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:15.667242 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:15.667204    4924 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 19:00:15.667495 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:15.667465    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx" podUID="53b2edae-97f1-44e9-9580-29d605357c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 19:00:15.704087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:15.703141    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5" podUID="925a4de9-b451-4263-9a1c-eebbbae4a08c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:15.732090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:15.706907    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 19:00:15.902869 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:15.902789    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 19:00:15.904450 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:15.904426    4924 scope.go:122] "RemoveContainer" containerID="54ba0e05e84fad67b4ce6977ad4e521d903c9545518a180fb34349d85dcbb1de"
Oct 30 19:00:15.904990 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:15.783667    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 19:00:15.943216 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:15.943167    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw" podUID="922dfc11-a4ba-4d50-a50f-b5155d277792" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 19:00:15.981154 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:15.940420    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m" podUID="1a90412d-c047-422e-b3c7-9546d286c303" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:15.985195 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:15.736447    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7" podUID="97a3b018-9083-4b7c-af88-6edddc585d67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:00:16.023348 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.014444    4924 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 19:00:16.023348 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.014512    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm" podUID="51837408-10f0-4026-b8c6-fcb002a1e459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 19:00:16.048348 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.033605    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg" podUID="9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:16.176562 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.090255    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd" podUID="03ac954a-d07b-4523-8d5a-a13dffd0bd2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:00:16.178634 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.178608    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 19:00:16.198644 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.198599    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld" podUID="f04af09f-b8d2-4b47-b0b3-88eb12adff93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:16.201118 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.201092    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 19:00:16.352485 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.352437    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt" podUID="955e98b9-864b-4636-9511-8b69f981eaa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:16.376113 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.361213    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l" podUID="bdc8df4b-8c94-40b8-b4be-80d1432d87de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:16.376113 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.362468    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj" podUID="51f3e639-9cd4-4aaa-a568-f19b6d2704fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:16.376113 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.363506    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg" podUID="9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Oct 30 19:00:16.382084 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:00:16.378686    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:00:16.464748 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.385178    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m" podUID="1a90412d-c047-422e-b3c7-9546d286c303" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:16.504937 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.385213    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2" podUID="fdf4711c-418a-436b-848a-d46974566573" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:00:16.504937 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.385239    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd" podUID="a3d792df-177d-4f7a-bba8-f6351175462e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:16.504937 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.396144    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" podUID="eb82dd7d-e682-42a2-924e-a84f21172fc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:16.512190 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.511563    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 19:00:16.634242 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.634209    4924 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 19:00:16.634453 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.634428    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx" podUID="53b2edae-97f1-44e9-9580-29d605357c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 19:00:16.638896 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.638721    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86" podUID="60a58597-fdc4-4722-8d2a-29e475f84a76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:16.674086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.640542    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 19:00:16.676241 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.676195    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2" podUID="fdf4711c-418a-436b-848a-d46974566573" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:16.676505 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.676469    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" podUID="eb82dd7d-e682-42a2-924e-a84f21172fc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:16.710294 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.710264    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 19:00:16.770759 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.770657    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5" podUID="925a4de9-b451-4263-9a1c-eebbbae4a08c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:16.812775 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.812418    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 19:00:16.812775 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.812478    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw" podUID="922dfc11-a4ba-4d50-a50f-b5155d277792" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 19:00:16.876313 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:16.859155    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 19:00:17.024152 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:17.023197    4924 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 19:00:17.024152 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:17.023278    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm" podUID="51837408-10f0-4026-b8c6-fcb002a1e459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 19:00:17.274282 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:17.274182    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd" podUID="03ac954a-d07b-4523-8d5a-a13dffd0bd2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:00:17.587541 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:17.587444    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg" podUID="9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:17.592182 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:17.592155    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 19:00:17.595586 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:17.595569    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 19:00:17.595809 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:17.595798    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 19:00:17.595949 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:17.595939    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 19:00:17.596095 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:17.596085    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 19:00:17.596219 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:17.596210    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 19:00:17.596353 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:17.596343    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 19:00:17.597656 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:17.597641    4924 scope.go:122] "RemoveContainer" containerID="a1d45025556ad2a57c5b153599b50fc04b182cfd021443c687a2b831e0b87d19"
Oct 30 19:00:17.677090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:17.615722    4924 scope.go:122] "RemoveContainer" containerID="b85170ed608aaa3dba3e43804c4b6e301f53f24446bd535fb72fc0d2bc74b61d"
Oct 30 19:00:17.677373 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:17.640410    4924 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 19:00:17.677566 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:17.677536    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx" podUID="53b2edae-97f1-44e9-9580-29d605357c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 19:00:17.677717 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:17.677702    4924 scope.go:122] "RemoveContainer" containerID="8241af4014084cb2bebe86256e514a7dfaad45074f1b0d11009e8f5857d59491"
Oct 30 19:00:17.797583 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:17.796816    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 19:00:17.797583 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:17.797385    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 19:00:17.797583 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:17.797431    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw" podUID="922dfc11-a4ba-4d50-a50f-b5155d277792" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 19:00:17.990661 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:17.990630    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 19:00:17.993525 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:17.993497    4924 scope.go:122] "RemoveContainer" containerID="3772ebcc7753eb75d9de27de150abbcd7810ae3a9146d2775870b4db17871df0"
Oct 30 19:00:18.100657 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:18.096379    4924 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 19:00:18.100657 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:18.100206    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm" podUID="51837408-10f0-4026-b8c6-fcb002a1e459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 19:00:18.251651 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:18.243155    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-725a884d-nk6zp"]
Oct 30 19:00:18.293313 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:18.293286    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-725a884d-nk6zp"]
Oct 30 19:00:18.627600 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:18.627567    4924 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 19:00:18.627853 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:18.627794    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx" podUID="53b2edae-97f1-44e9-9580-29d605357c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 19:00:18.791243 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:18.791139    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 19:00:18.791243 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:18.791190    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw" podUID="922dfc11-a4ba-4d50-a50f-b5155d277792" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 19:00:18.989135 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:18.975244    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6720c96-91cf-48d3-8f67-41952fe0f88e" path="/var/lib/kubelet/pods/c6720c96-91cf-48d3-8f67-41952fe0f88e/volumes"
Oct 30 19:00:18.989135 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:18.985710    4924 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 19:00:18.989135 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:18.985763    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm" podUID="51837408-10f0-4026-b8c6-fcb002a1e459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 19:00:18.998862 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:18.997626    4924 generic.go:356] "Generic (PLEG): container finished" podID="56c52368-2c45-4ef9-aa51-d507aafee931" containerID="4c793a8b63f172ef24469ea144c428210501c8b0dda0b2f0111109b30270c102" exitCode=60
Oct 30 19:00:18.998862 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:18.997865    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd" event={"ID":"56c52368-2c45-4ef9-aa51-d507aafee931","Type":"ContainerDied","Data":"4c793a8b63f172ef24469ea144c428210501c8b0dda0b2f0111109b30270c102"}
Oct 30 19:00:18.998862 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:18.997921    4924 scope.go:122] "RemoveContainer" containerID="54ba0e05e84fad67b4ce6977ad4e521d903c9545518a180fb34349d85dcbb1de"
Oct 30 19:00:18.998862 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:18.998437    4924 scope.go:122] "RemoveContainer" containerID="4c793a8b63f172ef24469ea144c428210501c8b0dda0b2f0111109b30270c102"
Oct 30 19:00:18.998862 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:00:18.998788    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd_odh-ai-gateway-infra(56c52368-2c45-4ef9-aa51-d507aafee931)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd" podUID="56c52368-2c45-4ef9-aa51-d507aafee931"
Oct 30 19:00:19.684032 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:19.657287    4924 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 19:00:19.700109 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:19.696995    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx" podUID="53b2edae-97f1-44e9-9580-29d605357c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 19:00:19.795088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:19.786163    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 19:00:19.897174 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:19.894204    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw" podUID="922dfc11-a4ba-4d50-a50f-b5155d277792" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 19:00:19.999849 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:19.998657    4924 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 19:00:19.999849 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:19.998716    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm" podUID="51837408-10f0-4026-b8c6-fcb002a1e459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 19:00:20.029128 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:20.002812    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:20.399541 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:20.399502    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g" event={"ID":"7582ec87-a899-40e7-9fa8-793e9caaf2f2","Type":"ContainerStarted","Data":"90a8a8b8f244a857b90dcc705dc65f97f731ef3770fc4e52edf093da4ad58069"}
Oct 30 19:00:20.466158 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:20.457271    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc" event={"ID":"1ba14910-fb5d-40ec-bad0-39dfebd1506e","Type":"ContainerStarted","Data":"926ab9e0916fec7277af52f491835cd5bc3c589bd00a9b303990aa0e9df1168a"}
Oct 30 19:00:20.466158 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:20.457807    4924 scope.go:122] "RemoveContainer" containerID="926ab9e0916fec7277af52f491835cd5bc3c589bd00a9b303990aa0e9df1168a"
Oct 30 19:00:20.488998 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:00:20.483332    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc_odh-ai-gateway-infra(1ba14910-fb5d-40ec-bad0-39dfebd1506e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc" podUID="1ba14910-fb5d-40ec-bad0-39dfebd1506e"
Oct 30 19:00:20.497293 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:20.490899    4924 generic.go:356] "Generic (PLEG): container finished" podID="884e56e6-4b28-4245-a418-1782703a11df" containerID="886be12744cc93387e773c1505d5d237704aa0af8db299269a57837da7d33fba" exitCode=60
Oct 30 19:00:20.497293 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:20.491042    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl" event={"ID":"884e56e6-4b28-4245-a418-1782703a11df","Type":"ContainerDied","Data":"886be12744cc93387e773c1505d5d237704aa0af8db299269a57837da7d33fba"}
Oct 30 19:00:20.497293 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:20.491113    4924 scope.go:122] "RemoveContainer" containerID="8241af4014084cb2bebe86256e514a7dfaad45074f1b0d11009e8f5857d59491"
Oct 30 19:00:20.497293 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:20.491537    4924 scope.go:122] "RemoveContainer" containerID="886be12744cc93387e773c1505d5d237704aa0af8db299269a57837da7d33fba"
Oct 30 19:00:20.497293 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:00:20.491826    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl_odh-ai-gateway-infra(884e56e6-4b28-4245-a418-1782703a11df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl" podUID="884e56e6-4b28-4245-a418-1782703a11df"
Oct 30 19:00:20.499419 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:20.499390    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576" event={"ID":"c0f224d4-04cc-4493-b313-56626b0f50a1","Type":"ContainerStarted","Data":"85040904ac101ef14b03ada85c5685359d29b57cbdaf75904b397e09d3530432"}
Oct 30 19:00:20.499962 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:20.499942    4924 scope.go:122] "RemoveContainer" containerID="85040904ac101ef14b03ada85c5685359d29b57cbdaf75904b397e09d3530432"
Oct 30 19:00:20.500507 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:00:20.500483    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576_odh-ai-gateway-infra(c0f224d4-04cc-4493-b313-56626b0f50a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576" podUID="c0f224d4-04cc-4493-b313-56626b0f50a1"
Oct 30 19:00:20.647227 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:20.623440    4924 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 19:00:20.647227 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:20.623499    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx" podUID="53b2edae-97f1-44e9-9580-29d605357c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 19:00:20.771160 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:20.770399    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 19:00:20.771160 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:20.770462    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw" podUID="922dfc11-a4ba-4d50-a50f-b5155d277792" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 19:00:20.983192 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:20.983058    4924 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 19:00:20.983192 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:20.983136    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm" podUID="51837408-10f0-4026-b8c6-fcb002a1e459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 19:00:21.150753 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.150717    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd"]
Oct 30 19:00:21.186598 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.185647    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g" podStartSLOduration=21.185626564 podStartE2EDuration="21.185626564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:21.185376543 +0000 UTC m=+1422.462426988" watchObservedRunningTime="2026-10-30 19:00:21.185626564 +0000 UTC m=+1422.462677010"
Oct 30 19:00:21.437031 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.436932    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd"
Oct 30 19:00:21.564429 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.564384    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd"
Oct 30 19:00:21.564429 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.564413    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd" event={"ID":"56c52368-2c45-4ef9-aa51-d507aafee931","Type":"ContainerDied","Data":"de8f83d1a2fac28a4e4f746a7c81a10b8da5da54d84cbb4ba26aa513fb4fa61b"}
Oct 30 19:00:21.564774 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.564491    4924 scope.go:122] "RemoveContainer" containerID="4c793a8b63f172ef24469ea144c428210501c8b0dda0b2f0111109b30270c102"
Oct 30 19:00:21.568226 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.568151    4924 generic.go:356] "Generic (PLEG): container finished" podID="7582ec87-a899-40e7-9fa8-793e9caaf2f2" containerID="90a8a8b8f244a857b90dcc705dc65f97f731ef3770fc4e52edf093da4ad58069" exitCode=60
Oct 30 19:00:21.568362 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.568323    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g" event={"ID":"7582ec87-a899-40e7-9fa8-793e9caaf2f2","Type":"ContainerDied","Data":"90a8a8b8f244a857b90dcc705dc65f97f731ef3770fc4e52edf093da4ad58069"}
Oct 30 19:00:21.568815 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.568796    4924 scope.go:122] "RemoveContainer" containerID="90a8a8b8f244a857b90dcc705dc65f97f731ef3770fc4e52edf093da4ad58069"
Oct 30 19:00:21.569540 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:00:21.569513    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g_odh-ai-gateway-infra(7582ec87-a899-40e7-9fa8-793e9caaf2f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g" podUID="7582ec87-a899-40e7-9fa8-793e9caaf2f2"
Oct 30 19:00:21.571523 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.571485    4924 generic.go:356] "Generic (PLEG): container finished" podID="1ba14910-fb5d-40ec-bad0-39dfebd1506e" containerID="926ab9e0916fec7277af52f491835cd5bc3c589bd00a9b303990aa0e9df1168a" exitCode=60
Oct 30 19:00:21.571626 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.571584    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc" event={"ID":"1ba14910-fb5d-40ec-bad0-39dfebd1506e","Type":"ContainerDied","Data":"926ab9e0916fec7277af52f491835cd5bc3c589bd00a9b303990aa0e9df1168a"}
Oct 30 19:00:21.572490 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.571954    4924 scope.go:122] "RemoveContainer" containerID="926ab9e0916fec7277af52f491835cd5bc3c589bd00a9b303990aa0e9df1168a"
Oct 30 19:00:21.572490 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.572189    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56c52368-2c45-4ef9-aa51-d507aafee931-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56c52368-2c45-4ef9-aa51-d507aafee931-maas-api-service-ca\") pod \"56c52368-2c45-4ef9-aa51-d507aafee931\" (UID: \"56c52368-2c45-4ef9-aa51-d507aafee931\") "
Oct 30 19:00:21.572490 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.572236    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56c52368-2c45-4ef9-aa51-d507aafee931-kube-api-access-rgxdc\" (UniqueName: \"kubernetes.io/projected/56c52368-2c45-4ef9-aa51-d507aafee931-kube-api-access-rgxdc\") pod \"56c52368-2c45-4ef9-aa51-d507aafee931\" (UID: \"56c52368-2c45-4ef9-aa51-d507aafee931\") "
Oct 30 19:00:21.572490 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:00:21.572245    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc_odh-ai-gateway-infra(1ba14910-fb5d-40ec-bad0-39dfebd1506e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc" podUID="1ba14910-fb5d-40ec-bad0-39dfebd1506e"
Oct 30 19:00:21.572938 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.572909    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56c52368-2c45-4ef9-aa51-d507aafee931-maas-api-service-ca" pod "56c52368-2c45-4ef9-aa51-d507aafee931" (UID: "56c52368-2c45-4ef9-aa51-d507aafee931"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:21.573149 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.573116    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56c52368-2c45-4ef9-aa51-d507aafee931-maas-api-service-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:21.577193 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.577172    4924 generic.go:356] "Generic (PLEG): container finished" podID="c0f224d4-04cc-4493-b313-56626b0f50a1" containerID="85040904ac101ef14b03ada85c5685359d29b57cbdaf75904b397e09d3530432" exitCode=60
Oct 30 19:00:21.577292 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.577201    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576" event={"ID":"c0f224d4-04cc-4493-b313-56626b0f50a1","Type":"ContainerDied","Data":"85040904ac101ef14b03ada85c5685359d29b57cbdaf75904b397e09d3530432"}
Oct 30 19:00:21.577566 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.577541    4924 scope.go:122] "RemoveContainer" containerID="85040904ac101ef14b03ada85c5685359d29b57cbdaf75904b397e09d3530432"
Oct 30 19:00:21.577867 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:00:21.577794    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576_odh-ai-gateway-infra(c0f224d4-04cc-4493-b313-56626b0f50a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576" podUID="c0f224d4-04cc-4493-b313-56626b0f50a1"
Oct 30 19:00:21.596411 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.596358    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c52368-2c45-4ef9-aa51-d507aafee931-kube-api-access-rgxdc" pod "56c52368-2c45-4ef9-aa51-d507aafee931" (UID: "56c52368-2c45-4ef9-aa51-d507aafee931"). InnerVolumeSpecName "kube-api-access-rgxdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:21.609203 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.609031    4924 scope.go:122] "RemoveContainer" containerID="3772ebcc7753eb75d9de27de150abbcd7810ae3a9146d2775870b4db17871df0"
Oct 30 19:00:21.620726 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.620696    4924 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 19:00:21.620857 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.620748    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx" podUID="53b2edae-97f1-44e9-9580-29d605357c8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 19:00:21.641978 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.641959    4924 scope.go:122] "RemoveContainer" containerID="b85170ed608aaa3dba3e43804c4b6e301f53f24446bd535fb72fc0d2bc74b61d"
Oct 30 19:00:21.660724 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.660702    4924 scope.go:122] "RemoveContainer" containerID="a1d45025556ad2a57c5b153599b50fc04b182cfd021443c687a2b831e0b87d19"
Oct 30 19:00:21.675044 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.675009    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgxdc\" (UniqueName: \"kubernetes.io/projected/56c52368-2c45-4ef9-aa51-d507aafee931-kube-api-access-rgxdc\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:21.769696 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.769546    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 19:00:21.769696 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.769601    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw" podUID="922dfc11-a4ba-4d50-a50f-b5155d277792" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 19:00:21.778279 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.777835    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl"]
Oct 30 19:00:21.888818 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.888782    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc"]
Oct 30 19:00:21.988260 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.988223    4924 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body=
Oct 30 19:00:21.988739 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:21.988287    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm" podUID="51837408-10f0-4026-b8c6-fcb002a1e459" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused"
Oct 30 19:00:22.088787 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.088754    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl"
Oct 30 19:00:22.089240 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.089223    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576"]
Oct 30 19:00:22.175376 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.175339    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1091e678-zj4dk"]
Oct 30 19:00:22.176411 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.176373    4924 cpu_manager.go:414] "containerMap: removing container" podUID="27f97819-2d12-424c-b617-936dd6f9099b" containerName="revoke-keys"
Oct 30 19:00:22.176411 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.176410    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f97819-2d12-424c-b617-936dd6f9099b" containerName="revoke-keys"
Oct 30 19:00:22.176604 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.176442    4924 cpu_manager.go:414] "containerMap: removing container" podUID="884e56e6-4b28-4245-a418-1782703a11df" containerName="cleanup"
Oct 30 19:00:22.176604 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.176450    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884e56e6-4b28-4245-a418-1782703a11df" containerName="cleanup"
Oct 30 19:00:22.176604 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.176468    4924 cpu_manager.go:414] "containerMap: removing container" podUID="56c52368-2c45-4ef9-aa51-d507aafee931" containerName="cleanup"
Oct 30 19:00:22.176604 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.176478    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56c52368-2c45-4ef9-aa51-d507aafee931" containerName="cleanup"
Oct 30 19:00:22.176604 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.176498    4924 cpu_manager.go:414] "containerMap: removing container" podUID="884e56e6-4b28-4245-a418-1782703a11df" containerName="cleanup"
Oct 30 19:00:22.176604 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.176505    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884e56e6-4b28-4245-a418-1782703a11df" containerName="cleanup"
Oct 30 19:00:22.176604 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.176519    4924 cpu_manager.go:414] "containerMap: removing container" podUID="56c52368-2c45-4ef9-aa51-d507aafee931" containerName="cleanup"
Oct 30 19:00:22.176604 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.176526    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56c52368-2c45-4ef9-aa51-d507aafee931" containerName="cleanup"
Oct 30 19:00:22.176604 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.176556    4924 cpu_manager.go:414] "containerMap: removing container" podUID="56c52368-2c45-4ef9-aa51-d507aafee931" containerName="cleanup"
Oct 30 19:00:22.176604 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.176563    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56c52368-2c45-4ef9-aa51-d507aafee931" containerName="cleanup"
Oct 30 19:00:22.177164 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.176763    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f97819-2d12-424c-b617-936dd6f9099b" containerName="revoke-keys"
Oct 30 19:00:22.177164 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.176778    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="884e56e6-4b28-4245-a418-1782703a11df" containerName="cleanup"
Oct 30 19:00:22.177164 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.176788    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="56c52368-2c45-4ef9-aa51-d507aafee931" containerName="cleanup"
Oct 30 19:00:22.177164 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.176799    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="56c52368-2c45-4ef9-aa51-d507aafee931" containerName="cleanup"
Oct 30 19:00:22.177164 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.176811    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="56c52368-2c45-4ef9-aa51-d507aafee931" containerName="cleanup"
Oct 30 19:00:22.177164 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.176822    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="884e56e6-4b28-4245-a418-1782703a11df" containerName="cleanup"
Oct 30 19:00:22.181619 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.181598    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1091e678-zj4dk"
Oct 30 19:00:22.192933 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.192904    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884e56e6-4b28-4245-a418-1782703a11df-kube-api-access-lvktl\" (UniqueName: \"kubernetes.io/projected/884e56e6-4b28-4245-a418-1782703a11df-kube-api-access-lvktl\") pod \"884e56e6-4b28-4245-a418-1782703a11df\" (UID: \"884e56e6-4b28-4245-a418-1782703a11df\") "
Oct 30 19:00:22.193051 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.192997    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/884e56e6-4b28-4245-a418-1782703a11df-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/884e56e6-4b28-4245-a418-1782703a11df-maas-api-service-ca\") pod \"884e56e6-4b28-4245-a418-1782703a11df\" (UID: \"884e56e6-4b28-4245-a418-1782703a11df\") "
Oct 30 19:00:22.193619 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.193587    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884e56e6-4b28-4245-a418-1782703a11df-maas-api-service-ca" pod "884e56e6-4b28-4245-a418-1782703a11df" (UID: "884e56e6-4b28-4245-a418-1782703a11df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:22.194232 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.194207    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/884e56e6-4b28-4245-a418-1782703a11df-maas-api-service-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:22.197019 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.196989    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884e56e6-4b28-4245-a418-1782703a11df-kube-api-access-lvktl" pod "884e56e6-4b28-4245-a418-1782703a11df" (UID: "884e56e6-4b28-4245-a418-1782703a11df"). InnerVolumeSpecName "kube-api-access-lvktl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:22.202697 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.202669    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd"]
Oct 30 19:00:22.209984 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.209959    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g"]
Oct 30 19:00:22.216773 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.216748    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3b6ad6-29889780-5fhnd"]
Oct 30 19:00:22.233379 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.233340    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:00:22.237387 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.237348    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1091e678-zj4dk"]
Oct 30 19:00:22.295512 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.295419    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a843d47b-b14a-4052-8f9b-af32e3db80d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1091e678-zj4dk\" (UID: \"a843d47b-b14a-4052-8f9b-af32e3db80d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1091e678-zj4dk"
Oct 30 19:00:22.296100 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.296022    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a843d47b-b14a-4052-8f9b-af32e3db80d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1091e678-zj4dk\" (UID: \"a843d47b-b14a-4052-8f9b-af32e3db80d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1091e678-zj4dk"
Oct 30 19:00:22.296350 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.296330    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkwkj\" (UniqueName: \"kubernetes.io/projected/a843d47b-b14a-4052-8f9b-af32e3db80d7-kube-api-access-hkwkj\") pod \"maas-api-revoke-sub-1091e678-zj4dk\" (UID: \"a843d47b-b14a-4052-8f9b-af32e3db80d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1091e678-zj4dk"
Oct 30 19:00:22.296622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.296604    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvktl\" (UniqueName: \"kubernetes.io/projected/884e56e6-4b28-4245-a418-1782703a11df-kube-api-access-lvktl\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:22.398893 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.398845    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a843d47b-b14a-4052-8f9b-af32e3db80d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1091e678-zj4dk\" (UID: \"a843d47b-b14a-4052-8f9b-af32e3db80d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1091e678-zj4dk"
Oct 30 19:00:22.399093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.398925    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkwkj\" (UniqueName: \"kubernetes.io/projected/a843d47b-b14a-4052-8f9b-af32e3db80d7-kube-api-access-hkwkj\") pod \"maas-api-revoke-sub-1091e678-zj4dk\" (UID: \"a843d47b-b14a-4052-8f9b-af32e3db80d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1091e678-zj4dk"
Oct 30 19:00:22.399093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.399048    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a843d47b-b14a-4052-8f9b-af32e3db80d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1091e678-zj4dk\" (UID: \"a843d47b-b14a-4052-8f9b-af32e3db80d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1091e678-zj4dk"
Oct 30 19:00:22.401971 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.401944    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a843d47b-b14a-4052-8f9b-af32e3db80d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1091e678-zj4dk\" (UID: \"a843d47b-b14a-4052-8f9b-af32e3db80d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1091e678-zj4dk"
Oct 30 19:00:22.403876 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.403856    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a843d47b-b14a-4052-8f9b-af32e3db80d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1091e678-zj4dk\" (UID: \"a843d47b-b14a-4052-8f9b-af32e3db80d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1091e678-zj4dk"
Oct 30 19:00:22.485829 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.485791    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkwkj\" (UniqueName: \"kubernetes.io/projected/a843d47b-b14a-4052-8f9b-af32e3db80d7-kube-api-access-hkwkj\") pod \"maas-api-revoke-sub-1091e678-zj4dk\" (UID: \"a843d47b-b14a-4052-8f9b-af32e3db80d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1091e678-zj4dk"
Oct 30 19:00:22.526507 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.526470    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1091e678-zj4dk"
Oct 30 19:00:22.633103 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.629718    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl" event={"ID":"884e56e6-4b28-4245-a418-1782703a11df","Type":"ContainerDied","Data":"14045db139cc8b3ce68d3b272c844551f71b18de810e1882607f92e8ccd8730b"}
Oct 30 19:00:22.633103 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.629784    4924 scope.go:122] "RemoveContainer" containerID="886be12744cc93387e773c1505d5d237704aa0af8db299269a57837da7d33fba"
Oct 30 19:00:22.633103 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.629901    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl"
Oct 30 19:00:22.682092 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.680367    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:22.682092 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.681725    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:22.685831 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.683129    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:00:22.855231 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:22.854657    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56c52368-2c45-4ef9-aa51-d507aafee931" path="/var/lib/kubelet/pods/56c52368-2c45-4ef9-aa51-d507aafee931/volumes"
Oct 30 19:00:23.009605 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.006316    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576"
Oct 30 19:00:23.041593 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.041562    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0f224d4-04cc-4493-b313-56626b0f50a1-kube-api-access-d22w2\" (UniqueName: \"kubernetes.io/projected/c0f224d4-04cc-4493-b313-56626b0f50a1-kube-api-access-d22w2\") pod \"c0f224d4-04cc-4493-b313-56626b0f50a1\" (UID: \"c0f224d4-04cc-4493-b313-56626b0f50a1\") "
Oct 30 19:00:23.041763 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.041740    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0f224d4-04cc-4493-b313-56626b0f50a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0f224d4-04cc-4493-b313-56626b0f50a1-maas-api-service-ca\") pod \"c0f224d4-04cc-4493-b313-56626b0f50a1\" (UID: \"c0f224d4-04cc-4493-b313-56626b0f50a1\") "
Oct 30 19:00:23.043783 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.042821    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0f224d4-04cc-4493-b313-56626b0f50a1-maas-api-service-ca" pod "c0f224d4-04cc-4493-b313-56626b0f50a1" (UID: "c0f224d4-04cc-4493-b313-56626b0f50a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:23.046394 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.046373    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0f224d4-04cc-4493-b313-56626b0f50a1-kube-api-access-d22w2" pod "c0f224d4-04cc-4493-b313-56626b0f50a1" (UID: "c0f224d4-04cc-4493-b313-56626b0f50a1"). InnerVolumeSpecName "kube-api-access-d22w2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:23.051181 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.051157    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc"
Oct 30 19:00:23.061501 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.061440    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g"
Oct 30 19:00:23.069134 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.069115    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:23.073793 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.073776    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:23.075140 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.075122    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:00:23.143530 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.143493    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7582ec87-a899-40e7-9fa8-793e9caaf2f2-kube-api-access-2jxgm\" (UniqueName: \"kubernetes.io/projected/7582ec87-a899-40e7-9fa8-793e9caaf2f2-kube-api-access-2jxgm\") pod \"7582ec87-a899-40e7-9fa8-793e9caaf2f2\" (UID: \"7582ec87-a899-40e7-9fa8-793e9caaf2f2\") "
Oct 30 19:00:23.143759 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.143735    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7582ec87-a899-40e7-9fa8-793e9caaf2f2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7582ec87-a899-40e7-9fa8-793e9caaf2f2-maas-api-service-ca\") pod \"7582ec87-a899-40e7-9fa8-793e9caaf2f2\" (UID: \"7582ec87-a899-40e7-9fa8-793e9caaf2f2\") "
Oct 30 19:00:23.143860 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.143781    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba14910-fb5d-40ec-bad0-39dfebd1506e-kube-api-access-b9q79\" (UniqueName: \"kubernetes.io/projected/1ba14910-fb5d-40ec-bad0-39dfebd1506e-kube-api-access-b9q79\") pod \"1ba14910-fb5d-40ec-bad0-39dfebd1506e\" (UID: \"1ba14910-fb5d-40ec-bad0-39dfebd1506e\") "
Oct 30 19:00:23.143940 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.143923    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ba14910-fb5d-40ec-bad0-39dfebd1506e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ba14910-fb5d-40ec-bad0-39dfebd1506e-maas-api-service-ca\") pod \"1ba14910-fb5d-40ec-bad0-39dfebd1506e\" (UID: \"1ba14910-fb5d-40ec-bad0-39dfebd1506e\") "
Oct 30 19:00:23.144574 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.144544    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d22w2\" (UniqueName: \"kubernetes.io/projected/c0f224d4-04cc-4493-b313-56626b0f50a1-kube-api-access-d22w2\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:23.144702 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.144580    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0f224d4-04cc-4493-b313-56626b0f50a1-maas-api-service-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:23.151732 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.151700    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba14910-fb5d-40ec-bad0-39dfebd1506e-maas-api-service-ca" pod "1ba14910-fb5d-40ec-bad0-39dfebd1506e" (UID: "1ba14910-fb5d-40ec-bad0-39dfebd1506e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:23.152136 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.152107    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7582ec87-a899-40e7-9fa8-793e9caaf2f2-maas-api-service-ca" pod "7582ec87-a899-40e7-9fa8-793e9caaf2f2" (UID: "7582ec87-a899-40e7-9fa8-793e9caaf2f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:23.157568 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.157544    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7582ec87-a899-40e7-9fa8-793e9caaf2f2-kube-api-access-2jxgm" pod "7582ec87-a899-40e7-9fa8-793e9caaf2f2" (UID: "7582ec87-a899-40e7-9fa8-793e9caaf2f2"). InnerVolumeSpecName "kube-api-access-2jxgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:23.159019 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.158995    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba14910-fb5d-40ec-bad0-39dfebd1506e-kube-api-access-b9q79" pod "1ba14910-fb5d-40ec-bad0-39dfebd1506e" (UID: "1ba14910-fb5d-40ec-bad0-39dfebd1506e"). InnerVolumeSpecName "kube-api-access-b9q79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:23.245812 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.245780    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7582ec87-a899-40e7-9fa8-793e9caaf2f2-maas-api-service-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:23.245812 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.245810    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9q79\" (UniqueName: \"kubernetes.io/projected/1ba14910-fb5d-40ec-bad0-39dfebd1506e-kube-api-access-b9q79\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:23.245812 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.245823    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ba14910-fb5d-40ec-bad0-39dfebd1506e-maas-api-service-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:23.246036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.245833    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jxgm\" (UniqueName: \"kubernetes.io/projected/7582ec87-a899-40e7-9fa8-793e9caaf2f2-kube-api-access-2jxgm\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:23.280721 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:00:23.280690    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda843d47b_b14a_4052_8f9b_af32e3db80d7.slice/crio-49fcfb0ebf7846d27344169d483dde437fc652fcdda95fea55ab32c2100d2360 WatchSource:0}: Error finding container 49fcfb0ebf7846d27344169d483dde437fc652fcdda95fea55ab32c2100d2360: Status 404 returned error can't find the container with id 49fcfb0ebf7846d27344169d483dde437fc652fcdda95fea55ab32c2100d2360
Oct 30 19:00:23.281219 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.281195    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1091e678-zj4dk"]
Oct 30 19:00:23.322497 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.322467    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl"]
Oct 30 19:00:23.333311 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.333274    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-649d16-29889780-6vntl"]
Oct 30 19:00:23.336497 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.334782    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:23.336497 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.336486    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:23.338516 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.338495    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:00:23.666675 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.666640    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g"
Oct 30 19:00:23.666874 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.666648    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g" event={"ID":"7582ec87-a899-40e7-9fa8-793e9caaf2f2","Type":"ContainerDied","Data":"4836997fbe8551100a8a335aa060a3d6bc417927705b714c82cb188a8ddb0a62"}
Oct 30 19:00:23.666874 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.666834    4924 scope.go:122] "RemoveContainer" containerID="90a8a8b8f244a857b90dcc705dc65f97f731ef3770fc4e52edf093da4ad58069"
Oct 30 19:00:23.668764 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.668742    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc" event={"ID":"1ba14910-fb5d-40ec-bad0-39dfebd1506e","Type":"ContainerDied","Data":"6a717b9f4dc09eeecff4507ee78edb3d4a75531fd2ef941e46e22d4f11f7069d"}
Oct 30 19:00:23.668877 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.668846    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc"
Oct 30 19:00:23.679129 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.679098    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1091e678-zj4dk" event={"ID":"a843d47b-b14a-4052-8f9b-af32e3db80d7","Type":"ContainerStarted","Data":"bbad9d78ce4f774c2b601174d2ac179b21b9de2af5a238e9ec77fefdb5d0dfcb"}
Oct 30 19:00:23.679312 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.679287    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1091e678-zj4dk" event={"ID":"a843d47b-b14a-4052-8f9b-af32e3db80d7","Type":"ContainerStarted","Data":"49fcfb0ebf7846d27344169d483dde437fc652fcdda95fea55ab32c2100d2360"}
Oct 30 19:00:23.681104 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.681048    4924 scope.go:122] "RemoveContainer" containerID="926ab9e0916fec7277af52f491835cd5bc3c589bd00a9b303990aa0e9df1168a"
Oct 30 19:00:23.684440 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.684376    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576"
Oct 30 19:00:23.684440 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.684409    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576" event={"ID":"c0f224d4-04cc-4493-b313-56626b0f50a1","Type":"ContainerDied","Data":"3a85787b240e2f5ad3e10331186e36bace9443f93035180a6e9a3e15e91966c1"}
Oct 30 19:00:23.694012 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.693996    4924 scope.go:122] "RemoveContainer" containerID="85040904ac101ef14b03ada85c5685359d29b57cbdaf75904b397e09d3530432"
Oct 30 19:00:23.767131 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.767097    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba23f01-627tz"]
Oct 30 19:00:23.768099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.768058    4924 cpu_manager.go:414] "containerMap: removing container" podUID="c0f224d4-04cc-4493-b313-56626b0f50a1" containerName="cleanup"
Oct 30 19:00:23.768099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.768098    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0f224d4-04cc-4493-b313-56626b0f50a1" containerName="cleanup"
Oct 30 19:00:23.768278 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.768125    4924 cpu_manager.go:414] "containerMap: removing container" podUID="7582ec87-a899-40e7-9fa8-793e9caaf2f2" containerName="cleanup"
Oct 30 19:00:23.768278 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.768133    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7582ec87-a899-40e7-9fa8-793e9caaf2f2" containerName="cleanup"
Oct 30 19:00:23.768278 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.768153    4924 cpu_manager.go:414] "containerMap: removing container" podUID="1ba14910-fb5d-40ec-bad0-39dfebd1506e" containerName="cleanup"
Oct 30 19:00:23.768278 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.768160    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ba14910-fb5d-40ec-bad0-39dfebd1506e" containerName="cleanup"
Oct 30 19:00:23.768278 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.768187    4924 cpu_manager.go:414] "containerMap: removing container" podUID="884e56e6-4b28-4245-a418-1782703a11df" containerName="cleanup"
Oct 30 19:00:23.768278 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.768194    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884e56e6-4b28-4245-a418-1782703a11df" containerName="cleanup"
Oct 30 19:00:23.768278 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.768218    4924 cpu_manager.go:414] "containerMap: removing container" podUID="c0f224d4-04cc-4493-b313-56626b0f50a1" containerName="cleanup"
Oct 30 19:00:23.768278 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.768225    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0f224d4-04cc-4493-b313-56626b0f50a1" containerName="cleanup"
Oct 30 19:00:23.768278 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.768253    4924 cpu_manager.go:414] "containerMap: removing container" podUID="1ba14910-fb5d-40ec-bad0-39dfebd1506e" containerName="cleanup"
Oct 30 19:00:23.768278 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.768260    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ba14910-fb5d-40ec-bad0-39dfebd1506e" containerName="cleanup"
Oct 30 19:00:23.768278 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.768275    4924 cpu_manager.go:414] "containerMap: removing container" podUID="7582ec87-a899-40e7-9fa8-793e9caaf2f2" containerName="cleanup"
Oct 30 19:00:23.768278 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.768282    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7582ec87-a899-40e7-9fa8-793e9caaf2f2" containerName="cleanup"
Oct 30 19:00:23.768850 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.768439    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="7582ec87-a899-40e7-9fa8-793e9caaf2f2" containerName="cleanup"
Oct 30 19:00:23.768850 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.768454    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ba14910-fb5d-40ec-bad0-39dfebd1506e" containerName="cleanup"
Oct 30 19:00:23.768850 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.768463    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="884e56e6-4b28-4245-a418-1782703a11df" containerName="cleanup"
Oct 30 19:00:23.768850 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.768477    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0f224d4-04cc-4493-b313-56626b0f50a1" containerName="cleanup"
Oct 30 19:00:23.768850 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.768490    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0f224d4-04cc-4493-b313-56626b0f50a1" containerName="cleanup"
Oct 30 19:00:23.768850 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.768503    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="7582ec87-a899-40e7-9fa8-793e9caaf2f2" containerName="cleanup"
Oct 30 19:00:23.768850 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.768514    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ba14910-fb5d-40ec-bad0-39dfebd1506e" containerName="cleanup"
Oct 30 19:00:23.768850 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.768522    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="7582ec87-a899-40e7-9fa8-793e9caaf2f2" containerName="cleanup"
Oct 30 19:00:23.774258 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.774237    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba23f01-627tz"
Oct 30 19:00:23.825578 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.825550    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba23f01-627tz"]
Oct 30 19:00:23.854636 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.854607    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576"]
Oct 30 19:00:23.858971 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.858948    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889780-vz576"]
Oct 30 19:00:23.859331 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.859305    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glt2z\" (UniqueName: \"kubernetes.io/projected/c9221c05-4591-4b67-b3d3-ccd6bddcff75-kube-api-access-glt2z\") pod \"maas-api-revoke-sub-fba23f01-627tz\" (UID: \"c9221c05-4591-4b67-b3d3-ccd6bddcff75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba23f01-627tz"
Oct 30 19:00:23.859387 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.859364    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9221c05-4591-4b67-b3d3-ccd6bddcff75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fba23f01-627tz\" (UID: \"c9221c05-4591-4b67-b3d3-ccd6bddcff75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba23f01-627tz"
Oct 30 19:00:23.859622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.859597    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9221c05-4591-4b67-b3d3-ccd6bddcff75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fba23f01-627tz\" (UID: \"c9221c05-4591-4b67-b3d3-ccd6bddcff75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba23f01-627tz"
Oct 30 19:00:23.960856 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.960784    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9221c05-4591-4b67-b3d3-ccd6bddcff75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fba23f01-627tz\" (UID: \"c9221c05-4591-4b67-b3d3-ccd6bddcff75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba23f01-627tz"
Oct 30 19:00:23.961005 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.960927    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glt2z\" (UniqueName: \"kubernetes.io/projected/c9221c05-4591-4b67-b3d3-ccd6bddcff75-kube-api-access-glt2z\") pod \"maas-api-revoke-sub-fba23f01-627tz\" (UID: \"c9221c05-4591-4b67-b3d3-ccd6bddcff75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba23f01-627tz"
Oct 30 19:00:23.961005 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.960956    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9221c05-4591-4b67-b3d3-ccd6bddcff75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fba23f01-627tz\" (UID: \"c9221c05-4591-4b67-b3d3-ccd6bddcff75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba23f01-627tz"
Oct 30 19:00:23.961453 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.961422    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9221c05-4591-4b67-b3d3-ccd6bddcff75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fba23f01-627tz\" (UID: \"c9221c05-4591-4b67-b3d3-ccd6bddcff75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba23f01-627tz"
Oct 30 19:00:23.963317 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.963294    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9221c05-4591-4b67-b3d3-ccd6bddcff75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fba23f01-627tz\" (UID: \"c9221c05-4591-4b67-b3d3-ccd6bddcff75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba23f01-627tz"
Oct 30 19:00:23.965382 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.965358    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g"]
Oct 30 19:00:23.973033 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:23.973012    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889780-r7v8g"]
Oct 30 19:00:24.015934 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:24.015907    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glt2z\" (UniqueName: \"kubernetes.io/projected/c9221c05-4591-4b67-b3d3-ccd6bddcff75-kube-api-access-glt2z\") pod \"maas-api-revoke-sub-fba23f01-627tz\" (UID: \"c9221c05-4591-4b67-b3d3-ccd6bddcff75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba23f01-627tz"
Oct 30 19:00:24.077813 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:24.077771    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1091e678-zj4dk" podStartSLOduration=2.077759885 podStartE2EDuration="2.077759885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:24.075175709 +0000 UTC m=+1425.352226158" watchObservedRunningTime="2026-10-30 19:00:24.077759885 +0000 UTC m=+1425.354810329"
Oct 30 19:00:24.092136 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:24.092116    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba23f01-627tz"
Oct 30 19:00:24.297728 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:24.297644    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc"]
Oct 30 19:00:24.302967 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:24.302940    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889780-7nbjc"]
Oct 30 19:00:24.467503 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:24.467462    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba23f01-627tz"]
Oct 30 19:00:24.467503 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:00:24.467488    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9221c05_4591_4b67_b3d3_ccd6bddcff75.slice/crio-a7911f607c726cdc26514904ea9199798e3ffb29a128253b2c4a811831b31d13 WatchSource:0}: Error finding container a7911f607c726cdc26514904ea9199798e3ffb29a128253b2c4a811831b31d13: Status 404 returned error can't find the container with id a7911f607c726cdc26514904ea9199798e3ffb29a128253b2c4a811831b31d13
Oct 30 19:00:24.713796 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:24.713765    4924 generic.go:356] "Generic (PLEG): container finished" podID="a843d47b-b14a-4052-8f9b-af32e3db80d7" containerID="bbad9d78ce4f774c2b601174d2ac179b21b9de2af5a238e9ec77fefdb5d0dfcb" exitCode=0
Oct 30 19:00:24.713958 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:24.713813    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1091e678-zj4dk" event={"ID":"a843d47b-b14a-4052-8f9b-af32e3db80d7","Type":"ContainerDied","Data":"bbad9d78ce4f774c2b601174d2ac179b21b9de2af5a238e9ec77fefdb5d0dfcb"}
Oct 30 19:00:24.716226 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:24.716194    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba23f01-627tz" event={"ID":"c9221c05-4591-4b67-b3d3-ccd6bddcff75","Type":"ContainerStarted","Data":"a7911f607c726cdc26514904ea9199798e3ffb29a128253b2c4a811831b31d13"}
Oct 30 19:00:24.838027 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:24.837990    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ba14910-fb5d-40ec-bad0-39dfebd1506e" path="/var/lib/kubelet/pods/1ba14910-fb5d-40ec-bad0-39dfebd1506e/volumes"
Oct 30 19:00:24.838572 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:24.838554    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7582ec87-a899-40e7-9fa8-793e9caaf2f2" path="/var/lib/kubelet/pods/7582ec87-a899-40e7-9fa8-793e9caaf2f2/volumes"
Oct 30 19:00:24.839050 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:24.839030    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="884e56e6-4b28-4245-a418-1782703a11df" path="/var/lib/kubelet/pods/884e56e6-4b28-4245-a418-1782703a11df/volumes"
Oct 30 19:00:24.839813 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:24.839797    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0f224d4-04cc-4493-b313-56626b0f50a1" path="/var/lib/kubelet/pods/c0f224d4-04cc-4493-b313-56626b0f50a1/volumes"
Oct 30 19:00:25.728014 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:25.727935    4924 generic.go:356] "Generic (PLEG): container finished" podID="c9221c05-4591-4b67-b3d3-ccd6bddcff75" containerID="85beb229203eaba8d32dfd0642927f00f25a3aaf7da9f0ed90168f7ba32c5563" exitCode=0
Oct 30 19:00:25.728014 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:25.727969    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba23f01-627tz" event={"ID":"c9221c05-4591-4b67-b3d3-ccd6bddcff75","Type":"ContainerDied","Data":"85beb229203eaba8d32dfd0642927f00f25a3aaf7da9f0ed90168f7ba32c5563"}
Oct 30 19:00:25.908324 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:25.908297    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1091e678-zj4dk"
Oct 30 19:00:25.989472 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:25.989388    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a843d47b-b14a-4052-8f9b-af32e3db80d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a843d47b-b14a-4052-8f9b-af32e3db80d7-maas-api-xks-ca\") pod \"a843d47b-b14a-4052-8f9b-af32e3db80d7\" (UID: \"a843d47b-b14a-4052-8f9b-af32e3db80d7\") "
Oct 30 19:00:25.989627 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:25.989508    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a843d47b-b14a-4052-8f9b-af32e3db80d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a843d47b-b14a-4052-8f9b-af32e3db80d7-maas-api-openshift-ca\") pod \"a843d47b-b14a-4052-8f9b-af32e3db80d7\" (UID: \"a843d47b-b14a-4052-8f9b-af32e3db80d7\") "
Oct 30 19:00:25.989627 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:25.989575    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a843d47b-b14a-4052-8f9b-af32e3db80d7-kube-api-access-hkwkj\" (UniqueName: \"kubernetes.io/projected/a843d47b-b14a-4052-8f9b-af32e3db80d7-kube-api-access-hkwkj\") pod \"a843d47b-b14a-4052-8f9b-af32e3db80d7\" (UID: \"a843d47b-b14a-4052-8f9b-af32e3db80d7\") "
Oct 30 19:00:25.989858 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:25.989832    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a843d47b-b14a-4052-8f9b-af32e3db80d7-maas-api-openshift-ca" pod "a843d47b-b14a-4052-8f9b-af32e3db80d7" (UID: "a843d47b-b14a-4052-8f9b-af32e3db80d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:25.990050 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:25.990036    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a843d47b-b14a-4052-8f9b-af32e3db80d7-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:25.991485 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:25.991460    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a843d47b-b14a-4052-8f9b-af32e3db80d7-maas-api-xks-ca" pod "a843d47b-b14a-4052-8f9b-af32e3db80d7" (UID: "a843d47b-b14a-4052-8f9b-af32e3db80d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:25.991598 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:25.991567    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a843d47b-b14a-4052-8f9b-af32e3db80d7-kube-api-access-hkwkj" pod "a843d47b-b14a-4052-8f9b-af32e3db80d7" (UID: "a843d47b-b14a-4052-8f9b-af32e3db80d7"). InnerVolumeSpecName "kube-api-access-hkwkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:26.091302 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:26.091266    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkwkj\" (UniqueName: \"kubernetes.io/projected/a843d47b-b14a-4052-8f9b-af32e3db80d7-kube-api-access-hkwkj\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:26.091302 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:26.091296    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a843d47b-b14a-4052-8f9b-af32e3db80d7-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:26.221043 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:26.221008    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:26.746757 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:26.746727    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1091e678-zj4dk"
Oct 30 19:00:26.746757 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:26.746740    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1091e678-zj4dk" event={"ID":"a843d47b-b14a-4052-8f9b-af32e3db80d7","Type":"ContainerDied","Data":"49fcfb0ebf7846d27344169d483dde437fc652fcdda95fea55ab32c2100d2360"}
Oct 30 19:00:26.747182 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:26.746809    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49fcfb0ebf7846d27344169d483dde437fc652fcdda95fea55ab32c2100d2360"
Oct 30 19:00:26.916193 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:26.916166    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba23f01-627tz"
Oct 30 19:00:26.929103 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:26.929045    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:27.003212 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:27.003129    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9221c05-4591-4b67-b3d3-ccd6bddcff75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9221c05-4591-4b67-b3d3-ccd6bddcff75-maas-api-xks-ca\") pod \"c9221c05-4591-4b67-b3d3-ccd6bddcff75\" (UID: \"c9221c05-4591-4b67-b3d3-ccd6bddcff75\") "
Oct 30 19:00:27.003375 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:27.003228    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9221c05-4591-4b67-b3d3-ccd6bddcff75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9221c05-4591-4b67-b3d3-ccd6bddcff75-maas-api-openshift-ca\") pod \"c9221c05-4591-4b67-b3d3-ccd6bddcff75\" (UID: \"c9221c05-4591-4b67-b3d3-ccd6bddcff75\") "
Oct 30 19:00:27.003375 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:27.003266    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9221c05-4591-4b67-b3d3-ccd6bddcff75-kube-api-access-glt2z\" (UniqueName: \"kubernetes.io/projected/c9221c05-4591-4b67-b3d3-ccd6bddcff75-kube-api-access-glt2z\") pod \"c9221c05-4591-4b67-b3d3-ccd6bddcff75\" (UID: \"c9221c05-4591-4b67-b3d3-ccd6bddcff75\") "
Oct 30 19:00:27.003563 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:27.003540    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9221c05-4591-4b67-b3d3-ccd6bddcff75-maas-api-openshift-ca" pod "c9221c05-4591-4b67-b3d3-ccd6bddcff75" (UID: "c9221c05-4591-4b67-b3d3-ccd6bddcff75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:27.003921 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:27.003898    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9221c05-4591-4b67-b3d3-ccd6bddcff75-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:27.005465 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:27.005440    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9221c05-4591-4b67-b3d3-ccd6bddcff75-maas-api-xks-ca" pod "c9221c05-4591-4b67-b3d3-ccd6bddcff75" (UID: "c9221c05-4591-4b67-b3d3-ccd6bddcff75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:27.005465 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:27.005456    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9221c05-4591-4b67-b3d3-ccd6bddcff75-kube-api-access-glt2z" pod "c9221c05-4591-4b67-b3d3-ccd6bddcff75" (UID: "c9221c05-4591-4b67-b3d3-ccd6bddcff75"). InnerVolumeSpecName "kube-api-access-glt2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:27.105115 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:27.105082    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glt2z\" (UniqueName: \"kubernetes.io/projected/c9221c05-4591-4b67-b3d3-ccd6bddcff75-kube-api-access-glt2z\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:27.105115 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:27.105112    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9221c05-4591-4b67-b3d3-ccd6bddcff75-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:27.756815 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:27.756780    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba23f01-627tz"
Oct 30 19:00:27.756815 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:27.756805    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba23f01-627tz" event={"ID":"c9221c05-4591-4b67-b3d3-ccd6bddcff75","Type":"ContainerDied","Data":"a7911f607c726cdc26514904ea9199798e3ffb29a128253b2c4a811831b31d13"}
Oct 30 19:00:27.757262 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:27.756856    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7911f607c726cdc26514904ea9199798e3ffb29a128253b2c4a811831b31d13"
Oct 30 19:00:31.151346 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:31.151308    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:32.898295 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:32.898260    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4332bcf9-j2qlr"]
Oct 30 19:00:32.899256 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:32.899230    4924 cpu_manager.go:414] "containerMap: removing container" podUID="1ba14910-fb5d-40ec-bad0-39dfebd1506e" containerName="cleanup"
Oct 30 19:00:32.899256 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:32.899257    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ba14910-fb5d-40ec-bad0-39dfebd1506e" containerName="cleanup"
Oct 30 19:00:32.899441 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:32.899275    4924 cpu_manager.go:414] "containerMap: removing container" podUID="7582ec87-a899-40e7-9fa8-793e9caaf2f2" containerName="cleanup"
Oct 30 19:00:32.899441 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:32.899285    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7582ec87-a899-40e7-9fa8-793e9caaf2f2" containerName="cleanup"
Oct 30 19:00:32.899441 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:32.899319    4924 cpu_manager.go:414] "containerMap: removing container" podUID="a843d47b-b14a-4052-8f9b-af32e3db80d7" containerName="revoke-keys"
Oct 30 19:00:32.899441 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:32.899327    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a843d47b-b14a-4052-8f9b-af32e3db80d7" containerName="revoke-keys"
Oct 30 19:00:32.899441 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:32.899352    4924 cpu_manager.go:414] "containerMap: removing container" podUID="c9221c05-4591-4b67-b3d3-ccd6bddcff75" containerName="revoke-keys"
Oct 30 19:00:32.899441 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:32.899359    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9221c05-4591-4b67-b3d3-ccd6bddcff75" containerName="revoke-keys"
Oct 30 19:00:32.899441 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:32.899394    4924 cpu_manager.go:414] "containerMap: removing container" podUID="c0f224d4-04cc-4493-b313-56626b0f50a1" containerName="cleanup"
Oct 30 19:00:32.899441 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:32.899402    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0f224d4-04cc-4493-b313-56626b0f50a1" containerName="cleanup"
Oct 30 19:00:32.899852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:32.899563    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ba14910-fb5d-40ec-bad0-39dfebd1506e" containerName="cleanup"
Oct 30 19:00:32.899852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:32.899583    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="a843d47b-b14a-4052-8f9b-af32e3db80d7" containerName="revoke-keys"
Oct 30 19:00:32.899852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:32.899595    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9221c05-4591-4b67-b3d3-ccd6bddcff75" containerName="revoke-keys"
Oct 30 19:00:32.899852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:32.899611    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0f224d4-04cc-4493-b313-56626b0f50a1" containerName="cleanup"
Oct 30 19:00:32.907094 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:32.904241    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4332bcf9-j2qlr"
Oct 30 19:00:32.956340 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:32.956312    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4332bcf9-j2qlr"]
Oct 30 19:00:32.990490 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:32.990461    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b34001f-f608-427e-ae9d-b304dcbea464-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4332bcf9-j2qlr\" (UID: \"5b34001f-f608-427e-ae9d-b304dcbea464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4332bcf9-j2qlr"
Oct 30 19:00:32.990634 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:32.990605    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g469\" (UniqueName: \"kubernetes.io/projected/5b34001f-f608-427e-ae9d-b304dcbea464-kube-api-access-6g469\") pod \"maas-api-revoke-sub-4332bcf9-j2qlr\" (UID: \"5b34001f-f608-427e-ae9d-b304dcbea464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4332bcf9-j2qlr"
Oct 30 19:00:32.990677 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:32.990666    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b34001f-f608-427e-ae9d-b304dcbea464-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4332bcf9-j2qlr\" (UID: \"5b34001f-f608-427e-ae9d-b304dcbea464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4332bcf9-j2qlr"
Oct 30 19:00:32.995543 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:32.995525    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:00:32.995711 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:32.995682    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:00:32.995788 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:32.995716    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:00:33.000671 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:33.000656    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:00:33.091904 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:33.091867    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g469\" (UniqueName: \"kubernetes.io/projected/5b34001f-f608-427e-ae9d-b304dcbea464-kube-api-access-6g469\") pod \"maas-api-revoke-sub-4332bcf9-j2qlr\" (UID: \"5b34001f-f608-427e-ae9d-b304dcbea464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4332bcf9-j2qlr"
Oct 30 19:00:33.091904 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:33.091911    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b34001f-f608-427e-ae9d-b304dcbea464-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4332bcf9-j2qlr\" (UID: \"5b34001f-f608-427e-ae9d-b304dcbea464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4332bcf9-j2qlr"
Oct 30 19:00:33.092149 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:33.092052    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b34001f-f608-427e-ae9d-b304dcbea464-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4332bcf9-j2qlr\" (UID: \"5b34001f-f608-427e-ae9d-b304dcbea464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4332bcf9-j2qlr"
Oct 30 19:00:33.093139 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:33.093107    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b34001f-f608-427e-ae9d-b304dcbea464-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4332bcf9-j2qlr\" (UID: \"5b34001f-f608-427e-ae9d-b304dcbea464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4332bcf9-j2qlr"
Oct 30 19:00:33.094368 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:33.094351    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b34001f-f608-427e-ae9d-b304dcbea464-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4332bcf9-j2qlr\" (UID: \"5b34001f-f608-427e-ae9d-b304dcbea464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4332bcf9-j2qlr"
Oct 30 19:00:33.147235 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:33.147208    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g469\" (UniqueName: \"kubernetes.io/projected/5b34001f-f608-427e-ae9d-b304dcbea464-kube-api-access-6g469\") pod \"maas-api-revoke-sub-4332bcf9-j2qlr\" (UID: \"5b34001f-f608-427e-ae9d-b304dcbea464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4332bcf9-j2qlr"
Oct 30 19:00:33.220559 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:33.220498    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4332bcf9-j2qlr"
Oct 30 19:00:33.262021 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:33.261988    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b97aee-fknpv"]
Oct 30 19:00:33.266864 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:33.266841    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b97aee-fknpv"
Oct 30 19:00:33.322124 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:33.322091    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b97aee-fknpv"]
Oct 30 19:00:33.397001 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:33.396969    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kjsm\" (UniqueName: \"kubernetes.io/projected/f3ef59b7-2567-49bb-ae3a-633348290767-kube-api-access-5kjsm\") pod \"maas-api-revoke-sub-d9b97aee-fknpv\" (UID: \"f3ef59b7-2567-49bb-ae3a-633348290767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b97aee-fknpv"
Oct 30 19:00:33.397205 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:33.397027    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3ef59b7-2567-49bb-ae3a-633348290767-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9b97aee-fknpv\" (UID: \"f3ef59b7-2567-49bb-ae3a-633348290767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b97aee-fknpv"
Oct 30 19:00:33.397205 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:33.397177    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3ef59b7-2567-49bb-ae3a-633348290767-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9b97aee-fknpv\" (UID: \"f3ef59b7-2567-49bb-ae3a-633348290767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b97aee-fknpv"
Oct 30 19:00:33.498759 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:33.498674    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kjsm\" (UniqueName: \"kubernetes.io/projected/f3ef59b7-2567-49bb-ae3a-633348290767-kube-api-access-5kjsm\") pod \"maas-api-revoke-sub-d9b97aee-fknpv\" (UID: \"f3ef59b7-2567-49bb-ae3a-633348290767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b97aee-fknpv"
Oct 30 19:00:33.498759 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:33.498741    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3ef59b7-2567-49bb-ae3a-633348290767-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9b97aee-fknpv\" (UID: \"f3ef59b7-2567-49bb-ae3a-633348290767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b97aee-fknpv"
Oct 30 19:00:33.498970 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:33.498832    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3ef59b7-2567-49bb-ae3a-633348290767-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9b97aee-fknpv\" (UID: \"f3ef59b7-2567-49bb-ae3a-633348290767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b97aee-fknpv"
Oct 30 19:00:33.499359 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:33.499337    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3ef59b7-2567-49bb-ae3a-633348290767-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9b97aee-fknpv\" (UID: \"f3ef59b7-2567-49bb-ae3a-633348290767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b97aee-fknpv"
Oct 30 19:00:33.501096 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:33.501059    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3ef59b7-2567-49bb-ae3a-633348290767-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9b97aee-fknpv\" (UID: \"f3ef59b7-2567-49bb-ae3a-633348290767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b97aee-fknpv"
Oct 30 19:00:33.553940 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:33.553909    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kjsm\" (UniqueName: \"kubernetes.io/projected/f3ef59b7-2567-49bb-ae3a-633348290767-kube-api-access-5kjsm\") pod \"maas-api-revoke-sub-d9b97aee-fknpv\" (UID: \"f3ef59b7-2567-49bb-ae3a-633348290767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b97aee-fknpv"
Oct 30 19:00:33.586948 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:33.586919    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b97aee-fknpv"
Oct 30 19:00:33.607103 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:00:33.606133    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b34001f_f608_427e_ae9d_b304dcbea464.slice/crio-d44818a5bcc917ed3a8d3228c40164452dc8e318b7bc470d97d6408db9dd155f WatchSource:0}: Error finding container d44818a5bcc917ed3a8d3228c40164452dc8e318b7bc470d97d6408db9dd155f: Status 404 returned error can't find the container with id d44818a5bcc917ed3a8d3228c40164452dc8e318b7bc470d97d6408db9dd155f
Oct 30 19:00:33.607298 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:33.607096    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4332bcf9-j2qlr"]
Oct 30 19:00:33.813185 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:33.813104    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4332bcf9-j2qlr" event={"ID":"5b34001f-f608-427e-ae9d-b304dcbea464","Type":"ContainerStarted","Data":"d44818a5bcc917ed3a8d3228c40164452dc8e318b7bc470d97d6408db9dd155f"}
Oct 30 19:00:33.992788 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:33.992764    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b97aee-fknpv"]
Oct 30 19:00:34.824890 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:34.824800    4924 generic.go:356] "Generic (PLEG): container finished" podID="5b34001f-f608-427e-ae9d-b304dcbea464" containerID="61b010d9e4fbc4c8c6b532645b547ba786c706b0a3af185c5e8c55b89b6ee4e9" exitCode=0
Oct 30 19:00:34.825046 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:34.824884    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4332bcf9-j2qlr" event={"ID":"5b34001f-f608-427e-ae9d-b304dcbea464","Type":"ContainerDied","Data":"61b010d9e4fbc4c8c6b532645b547ba786c706b0a3af185c5e8c55b89b6ee4e9"}
Oct 30 19:00:34.826593 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:34.826569    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b97aee-fknpv" event={"ID":"f3ef59b7-2567-49bb-ae3a-633348290767","Type":"ContainerStarted","Data":"807e68e10c71ab16adfaf75f8d14628b323478ccf5890a815442b3360b09fa64"}
Oct 30 19:00:34.835012 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:34.834989    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b97aee-fknpv" event={"ID":"f3ef59b7-2567-49bb-ae3a-633348290767","Type":"ContainerStarted","Data":"28ea34423090ffad79649ba4880fc2a7eeb6dde811a1893524b39cd1af56fdbf"}
Oct 30 19:00:35.042741 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:35.042686    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b97aee-fknpv" podStartSLOduration=2.042673613 podStartE2EDuration="2.042673613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:35.041022564 +0000 UTC m=+1436.318073009" watchObservedRunningTime="2026-10-30 19:00:35.042673613 +0000 UTC m=+1436.319724059"
Oct 30 19:00:35.839555 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:35.839525    4924 generic.go:356] "Generic (PLEG): container finished" podID="f3ef59b7-2567-49bb-ae3a-633348290767" containerID="807e68e10c71ab16adfaf75f8d14628b323478ccf5890a815442b3360b09fa64" exitCode=0
Oct 30 19:00:35.839756 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:35.839611    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b97aee-fknpv" event={"ID":"f3ef59b7-2567-49bb-ae3a-633348290767","Type":"ContainerDied","Data":"807e68e10c71ab16adfaf75f8d14628b323478ccf5890a815442b3360b09fa64"}
Oct 30 19:00:36.015022 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:36.014986    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:36.029259 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:36.029032    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4332bcf9-j2qlr"
Oct 30 19:00:36.139009 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:36.138861    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b34001f-f608-427e-ae9d-b304dcbea464-kube-api-access-6g469\" (UniqueName: \"kubernetes.io/projected/5b34001f-f608-427e-ae9d-b304dcbea464-kube-api-access-6g469\") pod \"5b34001f-f608-427e-ae9d-b304dcbea464\" (UID: \"5b34001f-f608-427e-ae9d-b304dcbea464\") "
Oct 30 19:00:36.139009 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:36.138937    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b34001f-f608-427e-ae9d-b304dcbea464-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b34001f-f608-427e-ae9d-b304dcbea464-maas-api-xks-ca\") pod \"5b34001f-f608-427e-ae9d-b304dcbea464\" (UID: \"5b34001f-f608-427e-ae9d-b304dcbea464\") "
Oct 30 19:00:36.139009 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:36.138995    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b34001f-f608-427e-ae9d-b304dcbea464-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b34001f-f608-427e-ae9d-b304dcbea464-maas-api-openshift-ca\") pod \"5b34001f-f608-427e-ae9d-b304dcbea464\" (UID: \"5b34001f-f608-427e-ae9d-b304dcbea464\") "
Oct 30 19:00:36.140730 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:36.140205    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b34001f-f608-427e-ae9d-b304dcbea464-maas-api-openshift-ca" pod "5b34001f-f608-427e-ae9d-b304dcbea464" (UID: "5b34001f-f608-427e-ae9d-b304dcbea464"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:36.141821 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:36.141795    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b34001f-f608-427e-ae9d-b304dcbea464-maas-api-xks-ca" pod "5b34001f-f608-427e-ae9d-b304dcbea464" (UID: "5b34001f-f608-427e-ae9d-b304dcbea464"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:36.152403 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:36.152377    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b34001f-f608-427e-ae9d-b304dcbea464-kube-api-access-6g469" pod "5b34001f-f608-427e-ae9d-b304dcbea464" (UID: "5b34001f-f608-427e-ae9d-b304dcbea464"). InnerVolumeSpecName "kube-api-access-6g469". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:36.240953 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:36.240919    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g469\" (UniqueName: \"kubernetes.io/projected/5b34001f-f608-427e-ae9d-b304dcbea464-kube-api-access-6g469\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:36.240953 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:36.240949    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b34001f-f608-427e-ae9d-b304dcbea464-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:36.240953 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:36.240959    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b34001f-f608-427e-ae9d-b304dcbea464-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:36.288155 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:36.288120    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:36.471131 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:36.471036    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:36.855053 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:36.855028    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4332bcf9-j2qlr"
Oct 30 19:00:36.855254 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:36.855085    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4332bcf9-j2qlr" event={"ID":"5b34001f-f608-427e-ae9d-b304dcbea464","Type":"ContainerDied","Data":"d44818a5bcc917ed3a8d3228c40164452dc8e318b7bc470d97d6408db9dd155f"}
Oct 30 19:00:36.855254 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:36.855155    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d44818a5bcc917ed3a8d3228c40164452dc8e318b7bc470d97d6408db9dd155f"
Oct 30 19:00:37.045845 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:37.045815    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b97aee-fknpv"
Oct 30 19:00:37.156721 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:37.156635    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3ef59b7-2567-49bb-ae3a-633348290767-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3ef59b7-2567-49bb-ae3a-633348290767-maas-api-openshift-ca\") pod \"f3ef59b7-2567-49bb-ae3a-633348290767\" (UID: \"f3ef59b7-2567-49bb-ae3a-633348290767\") "
Oct 30 19:00:37.175125 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:37.156835    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3ef59b7-2567-49bb-ae3a-633348290767-kube-api-access-5kjsm\" (UniqueName: \"kubernetes.io/projected/f3ef59b7-2567-49bb-ae3a-633348290767-kube-api-access-5kjsm\") pod \"f3ef59b7-2567-49bb-ae3a-633348290767\" (UID: \"f3ef59b7-2567-49bb-ae3a-633348290767\") "
Oct 30 19:00:37.175125 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:37.157135    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3ef59b7-2567-49bb-ae3a-633348290767-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3ef59b7-2567-49bb-ae3a-633348290767-maas-api-xks-ca\") pod \"f3ef59b7-2567-49bb-ae3a-633348290767\" (UID: \"f3ef59b7-2567-49bb-ae3a-633348290767\") "
Oct 30 19:00:37.186118 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:37.157385    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3ef59b7-2567-49bb-ae3a-633348290767-maas-api-openshift-ca" pod "f3ef59b7-2567-49bb-ae3a-633348290767" (UID: "f3ef59b7-2567-49bb-ae3a-633348290767"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:37.190089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:37.190032    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3ef59b7-2567-49bb-ae3a-633348290767-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:37.194783 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:37.193216    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3ef59b7-2567-49bb-ae3a-633348290767-kube-api-access-5kjsm" pod "f3ef59b7-2567-49bb-ae3a-633348290767" (UID: "f3ef59b7-2567-49bb-ae3a-633348290767"). InnerVolumeSpecName "kube-api-access-5kjsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:37.196228 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:37.196147    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3ef59b7-2567-49bb-ae3a-633348290767-maas-api-xks-ca" pod "f3ef59b7-2567-49bb-ae3a-633348290767" (UID: "f3ef59b7-2567-49bb-ae3a-633348290767"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:37.291820 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:37.291789    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kjsm\" (UniqueName: \"kubernetes.io/projected/f3ef59b7-2567-49bb-ae3a-633348290767-kube-api-access-5kjsm\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:37.291820 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:37.291817    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3ef59b7-2567-49bb-ae3a-633348290767-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:37.865316 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:37.865291    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b97aee-fknpv"
Oct 30 19:00:37.865498 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:37.865330    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b97aee-fknpv" event={"ID":"f3ef59b7-2567-49bb-ae3a-633348290767","Type":"ContainerDied","Data":"28ea34423090ffad79649ba4880fc2a7eeb6dde811a1893524b39cd1af56fdbf"}
Oct 30 19:00:37.865498 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:37.865385    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28ea34423090ffad79649ba4880fc2a7eeb6dde811a1893524b39cd1af56fdbf"
Oct 30 19:00:39.057932 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:39.057904    4924 scope.go:122] "RemoveContainer" containerID="414a5ced4b723aa7e8a5a55881774d1fd6e8a86e702399e7008b5281e1384ecd"
Oct 30 19:00:39.068819 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:39.068797    4924 scope.go:122] "RemoveContainer" containerID="67e40f64d31558d189f86f1ac9fa5464e6de924e3a44b8e54b9105a6439237e1"
Oct 30 19:00:39.081657 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:39.081636    4924 scope.go:122] "RemoveContainer" containerID="b4002e1d7184303f39a20d3df227fae164d4cf8db400ed8faeb6bbc6f3c4e8a3"
Oct 30 19:00:39.091177 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:39.091154    4924 scope.go:122] "RemoveContainer" containerID="1822e2665235bd60a5d1884412a098c4ebc3f021f5d44ab59baadf00a2df8dad"
Oct 30 19:00:39.100152 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:39.100133    4924 scope.go:122] "RemoveContainer" containerID="1e3d04566a88984835bb08093310ac996d1415a0262ea8cd3fa33739774252f7"
Oct 30 19:00:39.108961 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:39.108942    4924 scope.go:122] "RemoveContainer" containerID="3853bc6b174fe27b5ff888b539bdbf7d3537e8b09867ea2d1b338762bd367cc1"
Oct 30 19:00:39.117620 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:39.117601    4924 scope.go:122] "RemoveContainer" containerID="e784dd833e5ea8326da0b38950c04d2c4c50ae8006c9da704b3208176b2b556c"
Oct 30 19:00:40.772394 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:40.772361    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:43.977189 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:43.977150    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea593430-2gt89"]
Oct 30 19:00:43.977834 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:43.977818    4924 cpu_manager.go:414] "containerMap: removing container" podUID="f3ef59b7-2567-49bb-ae3a-633348290767" containerName="revoke-keys"
Oct 30 19:00:43.977896 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:43.977835    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3ef59b7-2567-49bb-ae3a-633348290767" containerName="revoke-keys"
Oct 30 19:00:43.977896 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:43.977865    4924 cpu_manager.go:414] "containerMap: removing container" podUID="5b34001f-f608-427e-ae9d-b304dcbea464" containerName="revoke-keys"
Oct 30 19:00:43.977896 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:43.977871    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b34001f-f608-427e-ae9d-b304dcbea464" containerName="revoke-keys"
Oct 30 19:00:43.977995 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:43.977973    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3ef59b7-2567-49bb-ae3a-633348290767" containerName="revoke-keys"
Oct 30 19:00:43.977995 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:43.977987    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b34001f-f608-427e-ae9d-b304dcbea464" containerName="revoke-keys"
Oct 30 19:00:43.983866 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:43.983849    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea593430-2gt89"
Oct 30 19:00:44.036728 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:44.036697    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea593430-2gt89"]
Oct 30 19:00:44.076699 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:44.076661    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:00:44.076699 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:44.076666    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:00:44.076882 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:44.076677    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:00:44.079684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:44.079668    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:00:44.089737 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:44.089714    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdcbv\" (UniqueName: \"kubernetes.io/projected/873be518-74f7-44f7-b547-5626b97ebfad-kube-api-access-tdcbv\") pod \"maas-api-revoke-sub-ea593430-2gt89\" (UID: \"873be518-74f7-44f7-b547-5626b97ebfad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea593430-2gt89"
Oct 30 19:00:44.090005 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:44.089960    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/873be518-74f7-44f7-b547-5626b97ebfad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea593430-2gt89\" (UID: \"873be518-74f7-44f7-b547-5626b97ebfad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea593430-2gt89"
Oct 30 19:00:44.090103 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:44.090055    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/873be518-74f7-44f7-b547-5626b97ebfad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea593430-2gt89\" (UID: \"873be518-74f7-44f7-b547-5626b97ebfad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea593430-2gt89"
Oct 30 19:00:44.191646 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:44.191610    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdcbv\" (UniqueName: \"kubernetes.io/projected/873be518-74f7-44f7-b547-5626b97ebfad-kube-api-access-tdcbv\") pod \"maas-api-revoke-sub-ea593430-2gt89\" (UID: \"873be518-74f7-44f7-b547-5626b97ebfad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea593430-2gt89"
Oct 30 19:00:44.191804 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:44.191776    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/873be518-74f7-44f7-b547-5626b97ebfad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea593430-2gt89\" (UID: \"873be518-74f7-44f7-b547-5626b97ebfad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea593430-2gt89"
Oct 30 19:00:44.191847 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:44.191819    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/873be518-74f7-44f7-b547-5626b97ebfad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea593430-2gt89\" (UID: \"873be518-74f7-44f7-b547-5626b97ebfad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea593430-2gt89"
Oct 30 19:00:44.192735 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:44.192709    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/873be518-74f7-44f7-b547-5626b97ebfad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea593430-2gt89\" (UID: \"873be518-74f7-44f7-b547-5626b97ebfad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea593430-2gt89"
Oct 30 19:00:44.194103 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:44.194077    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/873be518-74f7-44f7-b547-5626b97ebfad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea593430-2gt89\" (UID: \"873be518-74f7-44f7-b547-5626b97ebfad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea593430-2gt89"
Oct 30 19:00:44.246710 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:44.246652    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdcbv\" (UniqueName: \"kubernetes.io/projected/873be518-74f7-44f7-b547-5626b97ebfad-kube-api-access-tdcbv\") pod \"maas-api-revoke-sub-ea593430-2gt89\" (UID: \"873be518-74f7-44f7-b547-5626b97ebfad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea593430-2gt89"
Oct 30 19:00:44.297901 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:44.297737    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea593430-2gt89"
Oct 30 19:00:44.671053 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:00:44.671015    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod873be518_74f7_44f7_b547_5626b97ebfad.slice/crio-cce27396643fba0ab757ccbcd93bb5ba8fdb28154c772994a4c1e7637bc2bd2a WatchSource:0}: Error finding container cce27396643fba0ab757ccbcd93bb5ba8fdb28154c772994a4c1e7637bc2bd2a: Status 404 returned error can't find the container with id cce27396643fba0ab757ccbcd93bb5ba8fdb28154c772994a4c1e7637bc2bd2a
Oct 30 19:00:44.673284 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:44.673256    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea593430-2gt89"]
Oct 30 19:00:44.889129 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:44.889091    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f332265-2kkl4"]
Oct 30 19:00:44.895174 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:44.895154    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f332265-2kkl4"
Oct 30 19:00:44.931477 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:44.931442    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea593430-2gt89" event={"ID":"873be518-74f7-44f7-b547-5626b97ebfad","Type":"ContainerStarted","Data":"cce27396643fba0ab757ccbcd93bb5ba8fdb28154c772994a4c1e7637bc2bd2a"}
Oct 30 19:00:44.950055 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:44.950028    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f332265-2kkl4"]
Oct 30 19:00:45.010232 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:45.010009    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5ef146-ff29-46ed-b391-6c1962a4c6bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f332265-2kkl4\" (UID: \"ca5ef146-ff29-46ed-b391-6c1962a4c6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f332265-2kkl4"
Oct 30 19:00:45.010232 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:45.010089    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv8hc\" (UniqueName: \"kubernetes.io/projected/ca5ef146-ff29-46ed-b391-6c1962a4c6bf-kube-api-access-nv8hc\") pod \"maas-api-revoke-sub-4f332265-2kkl4\" (UID: \"ca5ef146-ff29-46ed-b391-6c1962a4c6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f332265-2kkl4"
Oct 30 19:00:45.010748 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:45.010372    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5ef146-ff29-46ed-b391-6c1962a4c6bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f332265-2kkl4\" (UID: \"ca5ef146-ff29-46ed-b391-6c1962a4c6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f332265-2kkl4"
Oct 30 19:00:45.112976 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:45.112932    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5ef146-ff29-46ed-b391-6c1962a4c6bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f332265-2kkl4\" (UID: \"ca5ef146-ff29-46ed-b391-6c1962a4c6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f332265-2kkl4"
Oct 30 19:00:45.113333 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:45.113311    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5ef146-ff29-46ed-b391-6c1962a4c6bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f332265-2kkl4\" (UID: \"ca5ef146-ff29-46ed-b391-6c1962a4c6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f332265-2kkl4"
Oct 30 19:00:45.113452 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:45.113362    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv8hc\" (UniqueName: \"kubernetes.io/projected/ca5ef146-ff29-46ed-b391-6c1962a4c6bf-kube-api-access-nv8hc\") pod \"maas-api-revoke-sub-4f332265-2kkl4\" (UID: \"ca5ef146-ff29-46ed-b391-6c1962a4c6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f332265-2kkl4"
Oct 30 19:00:45.114176 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:45.114144    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5ef146-ff29-46ed-b391-6c1962a4c6bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f332265-2kkl4\" (UID: \"ca5ef146-ff29-46ed-b391-6c1962a4c6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f332265-2kkl4"
Oct 30 19:00:45.115996 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:45.115973    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5ef146-ff29-46ed-b391-6c1962a4c6bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f332265-2kkl4\" (UID: \"ca5ef146-ff29-46ed-b391-6c1962a4c6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f332265-2kkl4"
Oct 30 19:00:45.168525 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:45.168480    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv8hc\" (UniqueName: \"kubernetes.io/projected/ca5ef146-ff29-46ed-b391-6c1962a4c6bf-kube-api-access-nv8hc\") pod \"maas-api-revoke-sub-4f332265-2kkl4\" (UID: \"ca5ef146-ff29-46ed-b391-6c1962a4c6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f332265-2kkl4"
Oct 30 19:00:45.217571 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:45.217469    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f332265-2kkl4"
Oct 30 19:00:45.658093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:45.631981    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:45.894237 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:45.894205    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f332265-2kkl4"]
Oct 30 19:00:46.010147 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:00:46.005713    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:00:46.021431 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:46.021391    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea593430-2gt89" event={"ID":"873be518-74f7-44f7-b547-5626b97ebfad","Type":"ContainerStarted","Data":"627a98df6325d22453182de899ad5d8976c021f15ec508aec78c48e9400198d8"}
Oct 30 19:00:46.035292 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:46.035256    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f332265-2kkl4" event={"ID":"ca5ef146-ff29-46ed-b391-6c1962a4c6bf","Type":"ContainerStarted","Data":"79f2da0ba541f78be1cf22e134dc7ea4a2d999a12365a0afed4de9341f2a60fe"}
Oct 30 19:00:46.280267 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:46.280139    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea593430-2gt89" podStartSLOduration=3.280118124 podStartE2EDuration="3.280118124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:46.217043002 +0000 UTC m=+1447.494093447" watchObservedRunningTime="2026-10-30 19:00:46.280118124 +0000 UTC m=+1447.557168570"
Oct 30 19:00:46.492678 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:46.492631    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m" podUID="1a90412d-c047-422e-b3c7-9546d286c303" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:00:46.493667 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:46.493633    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2" podUID="fdf4711c-418a-436b-848a-d46974566573" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:46.631036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:46.630193    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd" podUID="a3d792df-177d-4f7a-bba8-f6351175462e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:46.647517 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:46.647476    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld" podUID="f04af09f-b8d2-4b47-b0b3-88eb12adff93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:46.654219 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:46.654141    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2" podUID="fdf4711c-418a-436b-848a-d46974566573" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded"
Oct 30 19:00:46.679237 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:46.679181    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" podUID="eb82dd7d-e682-42a2-924e-a84f21172fc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:46.727090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:46.721354    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7" podUID="97a3b018-9083-4b7c-af88-6edddc585d67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:47.016090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:47.008348    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg" podUID="9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:47.157523 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:47.157483    4924 generic.go:356] "Generic (PLEG): container finished" podID="873be518-74f7-44f7-b547-5626b97ebfad" containerID="627a98df6325d22453182de899ad5d8976c021f15ec508aec78c48e9400198d8" exitCode=0
Oct 30 19:00:47.158245 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:47.158214    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea593430-2gt89" event={"ID":"873be518-74f7-44f7-b547-5626b97ebfad","Type":"ContainerDied","Data":"627a98df6325d22453182de899ad5d8976c021f15ec508aec78c48e9400198d8"}
Oct 30 19:00:47.167057 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:47.167022    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f332265-2kkl4" event={"ID":"ca5ef146-ff29-46ed-b391-6c1962a4c6bf","Type":"ContainerStarted","Data":"952a4c0f360c3fade3052485d828cff0caabb8a20c3461a4d269f4584334d42b"}
Oct 30 19:00:47.296234 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:47.284931    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd" podUID="03ac954a-d07b-4523-8d5a-a13dffd0bd2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:00:47.338921 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:47.333417    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt" podUID="955e98b9-864b-4636-9511-8b69f981eaa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:47.371089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:47.367181    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj" podUID="51f3e639-9cd4-4aaa-a568-f19b6d2704fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 19:00:47.371089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:47.367546    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l" podUID="bdc8df4b-8c94-40b8-b4be-80d1432d87de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:47.398119 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:47.395345    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" podUID="eb82dd7d-e682-42a2-924e-a84f21172fc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:47.488124 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:47.485191    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f332265-2kkl4" podStartSLOduration=3.485166358 podStartE2EDuration="3.485166358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:47.441777015 +0000 UTC m=+1448.718827460" watchObservedRunningTime="2026-10-30 19:00:47.485166358 +0000 UTC m=+1448.762216804"
Oct 30 19:00:47.546090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:47.542956    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2" podUID="fdf4711c-418a-436b-848a-d46974566573" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:47.546090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:47.543332    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m" podUID="1a90412d-c047-422e-b3c7-9546d286c303" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:00:47.546090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:47.543645    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 19:00:47.546090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:47.543705    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 19:00:47.653097 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:47.651168    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86" podUID="60a58597-fdc4-4722-8d2a-29e475f84a76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:47.653097 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:47.652120    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld" podUID="f04af09f-b8d2-4b47-b0b3-88eb12adff93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:47.653097 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:47.652406    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd" podUID="a3d792df-177d-4f7a-bba8-f6351175462e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded"
Oct 30 19:00:47.653097 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:47.652688    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 19:00:47.653097 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:47.652747    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 19:00:47.734079 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:47.732158    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7" podUID="97a3b018-9083-4b7c-af88-6edddc585d67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:47.788120 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:47.734330    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 19:00:47.835090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:47.794446    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5" podUID="925a4de9-b451-4263-9a1c-eebbbae4a08c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:00:47.989179 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:47.988137    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj" podUID="51f3e639-9cd4-4aaa-a568-f19b6d2704fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:48.043115 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:48.022098    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg" podUID="9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Oct 30 19:00:48.043115 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:48.022563    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 19:00:48.328137 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:48.300087    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd" podUID="03ac954a-d07b-4523-8d5a-a13dffd0bd2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:00:48.328137 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:48.300463    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 19:00:48.335619 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:48.334160    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt" podUID="955e98b9-864b-4636-9511-8b69f981eaa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:48.344582 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:48.343593    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 19:00:48.372092 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:48.370300    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l" podUID="bdc8df4b-8c94-40b8-b4be-80d1432d87de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:48.372092 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:48.371053    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj" podUID="51f3e639-9cd4-4aaa-a568-f19b6d2704fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 19:00:48.379112 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:48.373340    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 19:00:48.379112 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:48.373403    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 19:00:48.391097 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:48.390422    4924 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:00:48.391097 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:48.390474    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p" podUID="2eaee46c-23ef-410a-a5b6-450eba9711cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:48.416398 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:48.416358    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" podUID="eb82dd7d-e682-42a2-924e-a84f21172fc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:48.444799 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:48.444774    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 19:00:48.543200 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:48.543149    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2" podUID="fdf4711c-418a-436b-848a-d46974566573" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:48.543526 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:48.543498    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m" podUID="1a90412d-c047-422e-b3c7-9546d286c303" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:48.661608 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:48.655355    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86" podUID="60a58597-fdc4-4722-8d2a-29e475f84a76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:48.661608 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:48.656817    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd" podUID="a3d792df-177d-4f7a-bba8-f6351175462e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:48.661608 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:48.656901    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld" podUID="f04af09f-b8d2-4b47-b0b3-88eb12adff93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:00:48.695089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:48.681144    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 19:00:48.754180 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:48.754125    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7" podUID="97a3b018-9083-4b7c-af88-6edddc585d67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:48.804086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:48.799032    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5" podUID="925a4de9-b451-4263-9a1c-eebbbae4a08c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:48.804086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:48.800479    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86" podUID="60a58597-fdc4-4722-8d2a-29e475f84a76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded"
Oct 30 19:00:48.804086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:48.800617    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 19:00:48.981137 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:48.972981    4924 generic.go:356] "Generic (PLEG): container finished" podID="ca5ef146-ff29-46ed-b391-6c1962a4c6bf" containerID="952a4c0f360c3fade3052485d828cff0caabb8a20c3461a4d269f4584334d42b" exitCode=0
Oct 30 19:00:49.173116 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:49.160319    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg" podUID="9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:49.460090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:49.423308    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" podUID="eb82dd7d-e682-42a2-924e-a84f21172fc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 30 19:00:49.460090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:49.448358    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd" podUID="03ac954a-d07b-4523-8d5a-a13dffd0bd2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:00:49.460090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:49.454356    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt" podUID="955e98b9-864b-4636-9511-8b69f981eaa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 19:00:49.516301 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:49.505540    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj" podUID="51f3e639-9cd4-4aaa-a568-f19b6d2704fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 19:00:49.587209 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:49.562156    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2" podUID="fdf4711c-418a-436b-848a-d46974566573" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:00:49.895090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:49.572864    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l" podUID="bdc8df4b-8c94-40b8-b4be-80d1432d87de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Oct 30 19:00:49.895090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:49.665929    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86" podUID="60a58597-fdc4-4722-8d2a-29e475f84a76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:49.895090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:49.666141    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m" podUID="1a90412d-c047-422e-b3c7-9546d286c303" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 19:00:49.895090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:49.704133    4924 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:00:49.895090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:49.894654    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh" podUID="971fec4f-5a07-4eae-b318-0e9c8f381983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:49.895090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:49.807839    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5" podUID="925a4de9-b451-4263-9a1c-eebbbae4a08c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:00:49.895090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:49.806386    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7" podUID="97a3b018-9083-4b7c-af88-6edddc585d67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:00:49.895090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:49.807695    4924 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:00:49.895090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:49.894757    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz" podUID="467dd97e-420c-44ac-a249-17a3deacfb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:49.895090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:49.830960    4924 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:00:49.895090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:49.894797    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn" podUID="816f124c-a4bb-478e-b48a-c8b91a4113d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:50.043249 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:50.043198    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l" podUID="bdc8df4b-8c94-40b8-b4be-80d1432d87de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": context deadline exceeded"
Oct 30 19:00:50.071977 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:50.071918    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld" podUID="f04af09f-b8d2-4b47-b0b3-88eb12adff93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:00:50.090452 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:50.090414    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd" podUID="a3d792df-177d-4f7a-bba8-f6351175462e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:50.112465 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:00:50.112423    4924 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.261s"
Oct 30 19:00:50.114004 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:50.113976    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f332265-2kkl4" event={"ID":"ca5ef146-ff29-46ed-b391-6c1962a4c6bf","Type":"ContainerDied","Data":"952a4c0f360c3fade3052485d828cff0caabb8a20c3461a4d269f4584334d42b"}
Oct 30 19:00:50.308823 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:50.308723    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs" podUID="4d49737a-ecba-4943-a8c5-419f1a1f115a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:50.311250 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:50.311217    4924 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:00:50.311455 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:50.311431    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl" podUID="4947eb13-56e8-4d54-b253-21913852aeb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:50.349206 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:50.349168    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg" podUID="9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Oct 30 19:00:50.428414 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:50.428368    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" podUID="eb82dd7d-e682-42a2-924e-a84f21172fc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:50.536275 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:50.536222    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt" podUID="955e98b9-864b-4636-9511-8b69f981eaa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:50.538009 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:50.537968    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj" podUID="51f3e639-9cd4-4aaa-a568-f19b6d2704fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:50.538521 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:50.538495    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd" podUID="03ac954a-d07b-4523-8d5a-a13dffd0bd2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:00:50.716745 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:50.716706    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:00:50.717037 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:50.717008    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw" podUID="ccd81788-05eb-4696-938f-d6dc6d3ab2ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:50.919241 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:50.919187    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m" podUID="1a90412d-c047-422e-b3c7-9546d286c303" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:51.013150 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:50.976179    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2" podUID="fdf4711c-418a-436b-848a-d46974566573" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:51.013150 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:50.976898    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86" podUID="60a58597-fdc4-4722-8d2a-29e475f84a76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 19:00:51.013150 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:50.977614    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l" podUID="bdc8df4b-8c94-40b8-b4be-80d1432d87de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:51.139285 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.139176    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd" podUID="a3d792df-177d-4f7a-bba8-f6351175462e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/ready\": context deadline exceeded"
Oct 30 19:00:51.156627 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.156573    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld" podUID="f04af09f-b8d2-4b47-b0b3-88eb12adff93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:51.180536 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.172237    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5" podUID="925a4de9-b451-4263-9a1c-eebbbae4a08c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:00:51.527225 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.527181    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" podUID="eb82dd7d-e682-42a2-924e-a84f21172fc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 30 19:00:51.610083 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.609690    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg" podUID="9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:51.610585 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.610341    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt" podUID="955e98b9-864b-4636-9511-8b69f981eaa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 19:00:51.615775 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.610641    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj" podUID="51f3e639-9cd4-4aaa-a568-f19b6d2704fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded"
Oct 30 19:00:51.615775 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.610911    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd" podUID="03ac954a-d07b-4523-8d5a-a13dffd0bd2e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 19:00:51.652400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.648143    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:00:51.652400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.648203    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc" podUID="3d89616c-b699-44df-983a-d58cad14b601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:00:51.702450 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.700406    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7" podUID="97a3b018-9083-4b7c-af88-6edddc585d67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:00:51.855514 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:00:51.855484    4924 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.02s"
Oct 30 19:00:51.855673 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.855562    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 19:00:51.855761 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.855698    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 19:00:51.855761 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.855748    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 19:00:51.855869 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.855793    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 19:00:51.855869 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.855825    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 19:00:51.855985 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.855871    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 19:00:51.855985 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.855908    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 19:00:51.855985 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.855956    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 19:00:51.856144 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.855993    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 19:00:51.880748 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.880465    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 19:00:51.883314 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.881100    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 19:00:51.883314 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.882772    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea593430-2gt89" event={"ID":"873be518-74f7-44f7-b547-5626b97ebfad","Type":"ContainerDied","Data":"cce27396643fba0ab757ccbcd93bb5ba8fdb28154c772994a4c1e7637bc2bd2a"}
Oct 30 19:00:51.883314 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.882820    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cce27396643fba0ab757ccbcd93bb5ba8fdb28154c772994a4c1e7637bc2bd2a"
Oct 30 19:00:51.883314 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.883023    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd"
Oct 30 19:00:51.887563 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.887539    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea593430-2gt89"
Oct 30 19:00:51.904125 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.904094    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fdecb3b-zpc44"]
Oct 30 19:00:51.934824 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.934796    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 19:00:51.958974 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.958931    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fdecb3b-zpc44"]
Oct 30 19:00:51.978912 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.978022    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e280a1e-pjbp7"]
Oct 30 19:00:51.982190 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.982163    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f332265-2kkl4"
Oct 30 19:00:51.990220 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:51.990197    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e280a1e-pjbp7"]
Oct 30 19:00:52.038316 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:52.038284    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/873be518-74f7-44f7-b547-5626b97ebfad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/873be518-74f7-44f7-b547-5626b97ebfad-maas-api-openshift-ca\") pod \"873be518-74f7-44f7-b547-5626b97ebfad\" (UID: \"873be518-74f7-44f7-b547-5626b97ebfad\") "
Oct 30 19:00:52.038465 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:52.038341    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/873be518-74f7-44f7-b547-5626b97ebfad-kube-api-access-tdcbv\" (UniqueName: \"kubernetes.io/projected/873be518-74f7-44f7-b547-5626b97ebfad-kube-api-access-tdcbv\") pod \"873be518-74f7-44f7-b547-5626b97ebfad\" (UID: \"873be518-74f7-44f7-b547-5626b97ebfad\") "
Oct 30 19:00:52.038533 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:52.038484    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/873be518-74f7-44f7-b547-5626b97ebfad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/873be518-74f7-44f7-b547-5626b97ebfad-maas-api-xks-ca\") pod \"873be518-74f7-44f7-b547-5626b97ebfad\" (UID: \"873be518-74f7-44f7-b547-5626b97ebfad\") "
Oct 30 19:00:52.038673 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:52.038651    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/873be518-74f7-44f7-b547-5626b97ebfad-maas-api-openshift-ca" pod "873be518-74f7-44f7-b547-5626b97ebfad" (UID: "873be518-74f7-44f7-b547-5626b97ebfad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:52.040383 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:52.040355    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/873be518-74f7-44f7-b547-5626b97ebfad-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:52.040943 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:52.040923    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/873be518-74f7-44f7-b547-5626b97ebfad-kube-api-access-tdcbv" pod "873be518-74f7-44f7-b547-5626b97ebfad" (UID: "873be518-74f7-44f7-b547-5626b97ebfad"). InnerVolumeSpecName "kube-api-access-tdcbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:52.041015 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:52.040940    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/873be518-74f7-44f7-b547-5626b97ebfad-maas-api-xks-ca" pod "873be518-74f7-44f7-b547-5626b97ebfad" (UID: "873be518-74f7-44f7-b547-5626b97ebfad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:52.141104 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:52.140988    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca5ef146-ff29-46ed-b391-6c1962a4c6bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5ef146-ff29-46ed-b391-6c1962a4c6bf-maas-api-xks-ca\") pod \"ca5ef146-ff29-46ed-b391-6c1962a4c6bf\" (UID: \"ca5ef146-ff29-46ed-b391-6c1962a4c6bf\") "
Oct 30 19:00:52.141104 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:52.141037    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5ef146-ff29-46ed-b391-6c1962a4c6bf-kube-api-access-nv8hc\" (UniqueName: \"kubernetes.io/projected/ca5ef146-ff29-46ed-b391-6c1962a4c6bf-kube-api-access-nv8hc\") pod \"ca5ef146-ff29-46ed-b391-6c1962a4c6bf\" (UID: \"ca5ef146-ff29-46ed-b391-6c1962a4c6bf\") "
Oct 30 19:00:52.141360 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:52.141133    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca5ef146-ff29-46ed-b391-6c1962a4c6bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5ef146-ff29-46ed-b391-6c1962a4c6bf-maas-api-openshift-ca\") pod \"ca5ef146-ff29-46ed-b391-6c1962a4c6bf\" (UID: \"ca5ef146-ff29-46ed-b391-6c1962a4c6bf\") "
Oct 30 19:00:52.141561 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:52.141531    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca5ef146-ff29-46ed-b391-6c1962a4c6bf-maas-api-openshift-ca" pod "ca5ef146-ff29-46ed-b391-6c1962a4c6bf" (UID: "ca5ef146-ff29-46ed-b391-6c1962a4c6bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:52.141826 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:52.141804    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdcbv\" (UniqueName: \"kubernetes.io/projected/873be518-74f7-44f7-b547-5626b97ebfad-kube-api-access-tdcbv\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:52.141897 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:52.141833    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5ef146-ff29-46ed-b391-6c1962a4c6bf-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:52.141897 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:52.141848    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/873be518-74f7-44f7-b547-5626b97ebfad-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:52.143273 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:52.143252    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca5ef146-ff29-46ed-b391-6c1962a4c6bf-maas-api-xks-ca" pod "ca5ef146-ff29-46ed-b391-6c1962a4c6bf" (UID: "ca5ef146-ff29-46ed-b391-6c1962a4c6bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:52.143360 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:52.143331    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5ef146-ff29-46ed-b391-6c1962a4c6bf-kube-api-access-nv8hc" pod "ca5ef146-ff29-46ed-b391-6c1962a4c6bf" (UID: "ca5ef146-ff29-46ed-b391-6c1962a4c6bf"). InnerVolumeSpecName "kube-api-access-nv8hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:52.243405 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:52.243373    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5ef146-ff29-46ed-b391-6c1962a4c6bf-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:52.243405 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:52.243398    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv8hc\" (UniqueName: \"kubernetes.io/projected/ca5ef146-ff29-46ed-b391-6c1962a4c6bf-kube-api-access-nv8hc\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:52.836171 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:52.836092    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a267a84-7dc4-4098-a7d3-4b0e8fb1867e" path="/var/lib/kubelet/pods/9a267a84-7dc4-4098-a7d3-4b0e8fb1867e/volumes"
Oct 30 19:00:52.836625 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:52.836610    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a545e6e7-3f1f-42ed-8f96-147bef8d70c6" path="/var/lib/kubelet/pods/a545e6e7-3f1f-42ed-8f96-147bef8d70c6/volumes"
Oct 30 19:00:52.893960 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:52.893924    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f332265-2kkl4" event={"ID":"ca5ef146-ff29-46ed-b391-6c1962a4c6bf","Type":"ContainerDied","Data":"79f2da0ba541f78be1cf22e134dc7ea4a2d999a12365a0afed4de9341f2a60fe"}
Oct 30 19:00:52.894180 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:52.893989    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79f2da0ba541f78be1cf22e134dc7ea4a2d999a12365a0afed4de9341f2a60fe"
Oct 30 19:00:52.894180 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:52.893943    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f332265-2kkl4"
Oct 30 19:00:52.894301 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:52.894228    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea593430-2gt89"
Oct 30 19:00:53.736912 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:53.736873    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9799318d-zrb9d"]
Oct 30 19:00:53.738020 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:53.737990    4924 cpu_manager.go:414] "containerMap: removing container" podUID="ca5ef146-ff29-46ed-b391-6c1962a4c6bf" containerName="revoke-keys"
Oct 30 19:00:53.738020 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:53.738017    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca5ef146-ff29-46ed-b391-6c1962a4c6bf" containerName="revoke-keys"
Oct 30 19:00:53.738256 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:53.738093    4924 cpu_manager.go:414] "containerMap: removing container" podUID="873be518-74f7-44f7-b547-5626b97ebfad" containerName="revoke-keys"
Oct 30 19:00:53.738256 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:53.738102    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="873be518-74f7-44f7-b547-5626b97ebfad" containerName="revoke-keys"
Oct 30 19:00:53.738354 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:53.738322    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca5ef146-ff29-46ed-b391-6c1962a4c6bf" containerName="revoke-keys"
Oct 30 19:00:53.738354 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:53.738347    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="873be518-74f7-44f7-b547-5626b97ebfad" containerName="revoke-keys"
Oct 30 19:00:53.767839 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:53.767814    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9799318d-zrb9d"
Oct 30 19:00:53.792305 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:53.792269    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9799318d-zrb9d"]
Oct 30 19:00:53.820910 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:53.820881    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:00:53.821046 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:53.820964    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:00:53.821046 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:53.820978    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:00:53.821219 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:53.821199    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:00:53.873684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:53.873652    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fe3ad08-26ac-4e3d-868b-4a4e69706985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9799318d-zrb9d\" (UID: \"0fe3ad08-26ac-4e3d-868b-4a4e69706985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9799318d-zrb9d"
Oct 30 19:00:53.874044 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:53.873703    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jwh6\" (UniqueName: \"kubernetes.io/projected/0fe3ad08-26ac-4e3d-868b-4a4e69706985-kube-api-access-4jwh6\") pod \"maas-api-revoke-sub-9799318d-zrb9d\" (UID: \"0fe3ad08-26ac-4e3d-868b-4a4e69706985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9799318d-zrb9d"
Oct 30 19:00:53.874044 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:53.873889    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fe3ad08-26ac-4e3d-868b-4a4e69706985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9799318d-zrb9d\" (UID: \"0fe3ad08-26ac-4e3d-868b-4a4e69706985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9799318d-zrb9d"
Oct 30 19:00:53.975727 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:53.975695    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jwh6\" (UniqueName: \"kubernetes.io/projected/0fe3ad08-26ac-4e3d-868b-4a4e69706985-kube-api-access-4jwh6\") pod \"maas-api-revoke-sub-9799318d-zrb9d\" (UID: \"0fe3ad08-26ac-4e3d-868b-4a4e69706985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9799318d-zrb9d"
Oct 30 19:00:53.975910 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:53.975892    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fe3ad08-26ac-4e3d-868b-4a4e69706985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9799318d-zrb9d\" (UID: \"0fe3ad08-26ac-4e3d-868b-4a4e69706985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9799318d-zrb9d"
Oct 30 19:00:53.976147 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:53.976079    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fe3ad08-26ac-4e3d-868b-4a4e69706985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9799318d-zrb9d\" (UID: \"0fe3ad08-26ac-4e3d-868b-4a4e69706985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9799318d-zrb9d"
Oct 30 19:00:53.976842 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:53.976814    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fe3ad08-26ac-4e3d-868b-4a4e69706985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9799318d-zrb9d\" (UID: \"0fe3ad08-26ac-4e3d-868b-4a4e69706985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9799318d-zrb9d"
Oct 30 19:00:53.978969 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:53.978944    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fe3ad08-26ac-4e3d-868b-4a4e69706985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9799318d-zrb9d\" (UID: \"0fe3ad08-26ac-4e3d-868b-4a4e69706985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9799318d-zrb9d"
Oct 30 19:00:54.029803 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:54.029741    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jwh6\" (UniqueName: \"kubernetes.io/projected/0fe3ad08-26ac-4e3d-868b-4a4e69706985-kube-api-access-4jwh6\") pod \"maas-api-revoke-sub-9799318d-zrb9d\" (UID: \"0fe3ad08-26ac-4e3d-868b-4a4e69706985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9799318d-zrb9d"
Oct 30 19:00:54.084933 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:54.084898    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9799318d-zrb9d"
Oct 30 19:00:54.457904 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:54.457870    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:54.472445 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:00:54.472416    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe3ad08_26ac_4e3d_868b_4a4e69706985.slice/crio-acc84edefe398951e799c07df5fba479197c9cff4ab5c689bf891adb36255ce4 WatchSource:0}: Error finding container acc84edefe398951e799c07df5fba479197c9cff4ab5c689bf891adb36255ce4: Status 404 returned error can't find the container with id acc84edefe398951e799c07df5fba479197c9cff4ab5c689bf891adb36255ce4
Oct 30 19:00:54.474124 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:54.474095    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9799318d-zrb9d"]
Oct 30 19:00:54.915139 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:54.915091    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9799318d-zrb9d" event={"ID":"0fe3ad08-26ac-4e3d-868b-4a4e69706985","Type":"ContainerStarted","Data":"acc84edefe398951e799c07df5fba479197c9cff4ab5c689bf891adb36255ce4"}
Oct 30 19:00:55.925359 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:55.925328    4924 generic.go:356] "Generic (PLEG): container finished" podID="0fe3ad08-26ac-4e3d-868b-4a4e69706985" containerID="fa5e99019a7cd4614ba94f2bafa18dd25f1eeed29ef59a542f17bf4e833fe9b0" exitCode=0
Oct 30 19:00:55.925832 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:55.925414    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9799318d-zrb9d" event={"ID":"0fe3ad08-26ac-4e3d-868b-4a4e69706985","Type":"ContainerDied","Data":"fa5e99019a7cd4614ba94f2bafa18dd25f1eeed29ef59a542f17bf4e833fe9b0"}
Oct 30 19:00:56.884241 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:56.884211    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:57.073534 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:57.073505    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:57.099907 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:57.099886    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9799318d-zrb9d"
Oct 30 19:00:57.228390 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:57.228306    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fe3ad08-26ac-4e3d-868b-4a4e69706985-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fe3ad08-26ac-4e3d-868b-4a4e69706985-maas-api-openshift-ca\") pod \"0fe3ad08-26ac-4e3d-868b-4a4e69706985\" (UID: \"0fe3ad08-26ac-4e3d-868b-4a4e69706985\") "
Oct 30 19:00:57.228546 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:57.228434    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe3ad08-26ac-4e3d-868b-4a4e69706985-kube-api-access-4jwh6\" (UniqueName: \"kubernetes.io/projected/0fe3ad08-26ac-4e3d-868b-4a4e69706985-kube-api-access-4jwh6\") pod \"0fe3ad08-26ac-4e3d-868b-4a4e69706985\" (UID: \"0fe3ad08-26ac-4e3d-868b-4a4e69706985\") "
Oct 30 19:00:57.228546 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:57.228458    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fe3ad08-26ac-4e3d-868b-4a4e69706985-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fe3ad08-26ac-4e3d-868b-4a4e69706985-maas-api-xks-ca\") pod \"0fe3ad08-26ac-4e3d-868b-4a4e69706985\" (UID: \"0fe3ad08-26ac-4e3d-868b-4a4e69706985\") "
Oct 30 19:00:57.228768 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:57.228734    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fe3ad08-26ac-4e3d-868b-4a4e69706985-maas-api-openshift-ca" pod "0fe3ad08-26ac-4e3d-868b-4a4e69706985" (UID: "0fe3ad08-26ac-4e3d-868b-4a4e69706985"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:00:57.229267 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:57.229245    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fe3ad08-26ac-4e3d-868b-4a4e69706985-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:57.230665 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:57.230638    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fe3ad08-26ac-4e3d-868b-4a4e69706985-maas-api-xks-ca" pod "0fe3ad08-26ac-4e3d-868b-4a4e69706985" (UID: "0fe3ad08-26ac-4e3d-868b-4a4e69706985"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:57.231135 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:57.231107    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe3ad08-26ac-4e3d-868b-4a4e69706985-kube-api-access-4jwh6" pod "0fe3ad08-26ac-4e3d-868b-4a4e69706985" (UID: "0fe3ad08-26ac-4e3d-868b-4a4e69706985"). InnerVolumeSpecName "kube-api-access-4jwh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:57.330335 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:57.330299    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jwh6\" (UniqueName: \"kubernetes.io/projected/0fe3ad08-26ac-4e3d-868b-4a4e69706985-kube-api-access-4jwh6\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:57.330335 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:57.330330    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fe3ad08-26ac-4e3d-868b-4a4e69706985-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:57.697315 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:57.697282    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:57.942088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:57.942039    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9799318d-zrb9d"
Oct 30 19:00:57.942274 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:57.942102    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9799318d-zrb9d" event={"ID":"0fe3ad08-26ac-4e3d-868b-4a4e69706985","Type":"ContainerDied","Data":"acc84edefe398951e799c07df5fba479197c9cff4ab5c689bf891adb36255ce4"}
Oct 30 19:00:57.942274 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:57.942153    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acc84edefe398951e799c07df5fba479197c9cff4ab5c689bf891adb36255ce4"
Oct 30 19:00:58.028652 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.028621    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:58.248410 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.248341    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:58.542208 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.542119    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"]
Oct 30 19:00:58.543075 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.543048    4924 cpu_manager.go:414] "containerMap: removing container" podUID="0fe3ad08-26ac-4e3d-868b-4a4e69706985" containerName="revoke-keys"
Oct 30 19:00:58.543497 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.543080    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe3ad08-26ac-4e3d-868b-4a4e69706985" containerName="revoke-keys"
Oct 30 19:00:58.543497 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.543252    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe3ad08-26ac-4e3d-868b-4a4e69706985" containerName="revoke-keys"
Oct 30 19:00:58.547633 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.547612    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.602966 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.602917    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-948bc488-gw-openshift-default-dockercfg-4vdt7\""
Oct 30 19:00:58.649249 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.649219    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82102463-ed46-40e7-8c8f-becb671573e7-istio-token\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.649452 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.649334    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-istio-envoy\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.649452 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.649381    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-credential-socket\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.649452 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.649409    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-workload-socket\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.649452 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.649443    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82102463-ed46-40e7-8c8f-becb671573e7-istiod-ca-cert\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.649660 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.649611    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82102463-ed46-40e7-8c8f-becb671573e7-istio-podinfo\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.649796 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.649774    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpvmd\" (UniqueName: \"kubernetes.io/projected/82102463-ed46-40e7-8c8f-becb671573e7-kube-api-access-jpvmd\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.652181 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.652148    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-workload-certs\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.652298 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.652222    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-istio-data\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.657802 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.657777    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"]
Oct 30 19:00:58.688671 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.688609    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:58.754487 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.754454    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpvmd\" (UniqueName: \"kubernetes.io/projected/82102463-ed46-40e7-8c8f-becb671573e7-kube-api-access-jpvmd\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.754487 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.754487    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-workload-certs\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.754721 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.754509    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-istio-data\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.754721 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.754573    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82102463-ed46-40e7-8c8f-becb671573e7-istio-token\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.754721 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.754640    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-istio-envoy\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.754721 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.754667    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-credential-socket\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.754721 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.754687    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-workload-socket\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.754721 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.754705    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82102463-ed46-40e7-8c8f-becb671573e7-istiod-ca-cert\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.755045 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.754819    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82102463-ed46-40e7-8c8f-becb671573e7-istio-podinfo\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.755045 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.754911    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-workload-certs\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.755045 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.754958    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-istio-data\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.755045 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.755036    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-credential-socket\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.755278 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.755104    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-workload-socket\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.755442 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.755424    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82102463-ed46-40e7-8c8f-becb671573e7-istiod-ca-cert\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.757048 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.757027    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-istio-envoy\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.757385 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.757368    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82102463-ed46-40e7-8c8f-becb671573e7-istio-podinfo\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.809284 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.809201    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82102463-ed46-40e7-8c8f-becb671573e7-istio-token\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.811343 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.811268    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpvmd\" (UniqueName: \"kubernetes.io/projected/82102463-ed46-40e7-8c8f-becb671573e7-kube-api-access-jpvmd\") pod \"e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") " pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:58.865205 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:58.863936    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:00:59.206818 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:59.206778    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:59.353099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:59.353038    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"]
Oct 30 19:00:59.398289 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:00:59.398260    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82102463_ed46_40e7_8c8f_becb671573e7.slice/crio-e33578e9f34ade523d2583877b943971470e6ce9fb79dd5d9720ed461a9e40a2 WatchSource:0}: Error finding container e33578e9f34ade523d2583877b943971470e6ce9fb79dd5d9720ed461a9e40a2: Status 404 returned error can't find the container with id e33578e9f34ade523d2583877b943971470e6ce9fb79dd5d9720ed461a9e40a2
Oct 30 19:00:59.405095 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:59.401222    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:00:59.405095 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:59.401317    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:00:59.405095 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:59.401362    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:00:59.602245 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:59.602191    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:00:59.976911 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:59.976874    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v" event={"ID":"82102463-ed46-40e7-8c8f-becb671573e7","Type":"ContainerStarted","Data":"96c968c38085700c1c8e86121b29f7e5509fb9650ec2cebcac23adfd3c5d680c"}
Oct 30 19:00:59.977145 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:00:59.976932    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v" event={"ID":"82102463-ed46-40e7-8c8f-becb671573e7","Type":"ContainerStarted","Data":"e33578e9f34ade523d2583877b943971470e6ce9fb79dd5d9720ed461a9e40a2"}
Oct 30 19:01:00.436551 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:00.436502    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v" podStartSLOduration=2.436488753 podStartE2EDuration="2.436488753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:00.180268014 +0000 UTC m=+1461.457318455" watchObservedRunningTime="2026-10-30 19:01:00.436488753 +0000 UTC m=+1461.713539197"
Oct 30 19:01:00.437549 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:00.437526    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2953e0a1-t4l4q"]
Oct 30 19:01:00.442007 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:00.441992    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2953e0a1-t4l4q"
Oct 30 19:01:00.487992 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:00.487967    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2953e0a1-t4l4q\" (UID: \"d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2953e0a1-t4l4q"
Oct 30 19:01:00.488208 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:00.488188    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2b9w\" (UniqueName: \"kubernetes.io/projected/d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d-kube-api-access-b2b9w\") pod \"maas-api-revoke-sub-2953e0a1-t4l4q\" (UID: \"d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2953e0a1-t4l4q"
Oct 30 19:01:00.488299 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:00.488283    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2953e0a1-t4l4q\" (UID: \"d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2953e0a1-t4l4q"
Oct 30 19:01:00.492265 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:00.492245    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:01:00.492400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:00.492276    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:01:00.492400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:00.492287    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:01:00.492598 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:00.492323    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:01:00.495086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:00.495052    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2953e0a1-t4l4q"]
Oct 30 19:01:00.589599 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:00.589567    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2953e0a1-t4l4q\" (UID: \"d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2953e0a1-t4l4q"
Oct 30 19:01:00.589738 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:00.589725    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2953e0a1-t4l4q\" (UID: \"d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2953e0a1-t4l4q"
Oct 30 19:01:00.589948 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:00.589926    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2b9w\" (UniqueName: \"kubernetes.io/projected/d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d-kube-api-access-b2b9w\") pod \"maas-api-revoke-sub-2953e0a1-t4l4q\" (UID: \"d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2953e0a1-t4l4q"
Oct 30 19:01:00.590338 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:00.590318    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2953e0a1-t4l4q\" (UID: \"d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2953e0a1-t4l4q"
Oct 30 19:01:00.591995 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:00.591976    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2953e0a1-t4l4q\" (UID: \"d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2953e0a1-t4l4q"
Oct 30 19:01:00.643909 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:00.643878    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2b9w\" (UniqueName: \"kubernetes.io/projected/d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d-kube-api-access-b2b9w\") pod \"maas-api-revoke-sub-2953e0a1-t4l4q\" (UID: \"d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2953e0a1-t4l4q"
Oct 30 19:01:00.756797 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:00.756723    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2953e0a1-t4l4q"
Oct 30 19:01:00.865357 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:00.865313    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:01:00.873005 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:00.872226    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:01:00.985803 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:00.985772    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:01:00.986849 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:00.986823    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:01:01.051222 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:01.051161    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af4a17ad-z2fqm"]
Oct 30 19:01:01.055727 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:01.055704    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af4a17ad-z2fqm"]
Oct 30 19:01:01.129480 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:01.129450    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2953e0a1-t4l4q"]
Oct 30 19:01:01.130002 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:01:01.129963    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d7764c_082a_4bd9_aaf2_f1782bdf2a3d.slice/crio-d116ca70906d0774f2f29836c5eb73ebbd0cb426e4e7f279dfa5cb51dfe6709b WatchSource:0}: Error finding container d116ca70906d0774f2f29836c5eb73ebbd0cb426e4e7f279dfa5cb51dfe6709b: Status 404 returned error can't find the container with id d116ca70906d0774f2f29836c5eb73ebbd0cb426e4e7f279dfa5cb51dfe6709b
Oct 30 19:01:01.997408 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:01.997371    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2953e0a1-t4l4q" event={"ID":"d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d","Type":"ContainerStarted","Data":"e11f0a95a4afbca41bdf6d4edf3d1a90c0f59ca1baac2b92189d16353936c009"}
Oct 30 19:01:01.997774 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:01.997473    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2953e0a1-t4l4q" event={"ID":"d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d","Type":"ContainerStarted","Data":"d116ca70906d0774f2f29836c5eb73ebbd0cb426e4e7f279dfa5cb51dfe6709b"}
Oct 30 19:01:02.564399 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:02.564364    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:02.763645 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:02.763610    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:02.836900 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:02.836824    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0db98356-328c-4cb0-8641-2d7d88f38d96" path="/var/lib/kubelet/pods/0db98356-328c-4cb0-8641-2d7d88f38d96/volumes"
Oct 30 19:01:03.007255 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:03.007222    4924 generic.go:356] "Generic (PLEG): container finished" podID="d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d" containerID="e11f0a95a4afbca41bdf6d4edf3d1a90c0f59ca1baac2b92189d16353936c009" exitCode=0
Oct 30 19:01:03.007740 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:03.007303    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2953e0a1-t4l4q" event={"ID":"d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d","Type":"ContainerDied","Data":"e11f0a95a4afbca41bdf6d4edf3d1a90c0f59ca1baac2b92189d16353936c009"}
Oct 30 19:01:03.169848 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:03.169823    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2953e0a1-t4l4q"
Oct 30 19:01:03.226948 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:03.226913    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d-maas-api-xks-ca\") pod \"d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d\" (UID: \"d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d\") "
Oct 30 19:01:03.227151 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:03.227112    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d-maas-api-openshift-ca\") pod \"d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d\" (UID: \"d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d\") "
Oct 30 19:01:03.227241 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:03.227154    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d-kube-api-access-b2b9w\" (UniqueName: \"kubernetes.io/projected/d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d-kube-api-access-b2b9w\") pod \"d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d\" (UID: \"d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d\") "
Oct 30 19:01:03.227509 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:03.227485    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d-maas-api-openshift-ca" pod "d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d" (UID: "d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:03.227725 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:03.227708    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:03.229084 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:03.229045    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d-maas-api-xks-ca" pod "d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d" (UID: "d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:03.229202 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:03.229178    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d-kube-api-access-b2b9w" pod "d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d" (UID: "d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d"). InnerVolumeSpecName "kube-api-access-b2b9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:03.328889 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:03.328845    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2b9w\" (UniqueName: \"kubernetes.io/projected/d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d-kube-api-access-b2b9w\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:03.328889 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:03.328879    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:04.017827 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:04.017800    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2953e0a1-t4l4q"
Oct 30 19:01:04.018366 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:04.017843    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2953e0a1-t4l4q" event={"ID":"d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d","Type":"ContainerDied","Data":"d116ca70906d0774f2f29836c5eb73ebbd0cb426e4e7f279dfa5cb51dfe6709b"}
Oct 30 19:01:04.018366 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:04.017898    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d116ca70906d0774f2f29836c5eb73ebbd0cb426e4e7f279dfa5cb51dfe6709b"
Oct 30 19:01:04.371988 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:04.371957    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:04.547256 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:04.547224    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:04.730920 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:04.730836    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:04.913716 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:04.913676    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:05.115958 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.115921    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:05.271331 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.271297    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn"]
Oct 30 19:01:05.271989 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.271973    4924 cpu_manager.go:414] "containerMap: removing container" podUID="d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d" containerName="revoke-keys"
Oct 30 19:01:05.272038 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.271991    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d" containerName="revoke-keys"
Oct 30 19:01:05.272135 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.272124    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d" containerName="revoke-keys"
Oct 30 19:01:05.275963 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.275948    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn"
Oct 30 19:01:05.318979 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.318953    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:05.332759 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.332732    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn"]
Oct 30 19:01:05.362715 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.362676    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hh5p\" (UniqueName: \"kubernetes.io/projected/4d01e582-b7f9-40a0-b166-13474d7f263c-kube-api-access-5hh5p\") pod \"payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn\" (UID: \"4d01e582-b7f9-40a0-b166-13474d7f263c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn"
Oct 30 19:01:05.362860 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.362794    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d01e582-b7f9-40a0-b166-13474d7f263c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn\" (UID: \"4d01e582-b7f9-40a0-b166-13474d7f263c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn"
Oct 30 19:01:05.371942 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.371877    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-948bc488\""
Oct 30 19:01:05.372174 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.372150    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-948bc488-dockercfg-gzt7j\""
Oct 30 19:01:05.380347 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.380321    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"]
Oct 30 19:01:05.385876 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.385859    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.439307 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.439282    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-85b6ee7b-openshift-default-dockercfg-xrpnp\""
Oct 30 19:01:05.440823 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.440798    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"]
Oct 30 19:01:05.464478 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.464450    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-token\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.464599 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.464519    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-data\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.464668 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.464622    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-credential-socket\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.464727 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.464686    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istiod-ca-cert\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.464848 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.464833    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-envoy\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.464912 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.464871    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-podinfo\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.464982 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.464965    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsjnz\" (UniqueName: \"kubernetes.io/projected/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-kube-api-access-wsjnz\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.465039 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.465027    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-workload-socket\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.465121 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.465080    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hh5p\" (UniqueName: \"kubernetes.io/projected/4d01e582-b7f9-40a0-b166-13474d7f263c-kube-api-access-5hh5p\") pod \"payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn\" (UID: \"4d01e582-b7f9-40a0-b166-13474d7f263c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn"
Oct 30 19:01:05.465201 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.465180    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d01e582-b7f9-40a0-b166-13474d7f263c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn\" (UID: \"4d01e582-b7f9-40a0-b166-13474d7f263c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn"
Oct 30 19:01:05.465255 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.465212    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-workload-certs\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.466152 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.466128    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d01e582-b7f9-40a0-b166-13474d7f263c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn\" (UID: \"4d01e582-b7f9-40a0-b166-13474d7f263c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn"
Oct 30 19:01:05.519144 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.519117    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hh5p\" (UniqueName: \"kubernetes.io/projected/4d01e582-b7f9-40a0-b166-13474d7f263c-kube-api-access-5hh5p\") pod \"payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn\" (UID: \"4d01e582-b7f9-40a0-b166-13474d7f263c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn"
Oct 30 19:01:05.521995 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.521966    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:05.577420 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.577382    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-envoy\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.577688 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.577669    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-podinfo\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.577838 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.577820    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsjnz\" (UniqueName: \"kubernetes.io/projected/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-kube-api-access-wsjnz\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.577990 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.577975    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-workload-socket\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.578506 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.578483    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-workload-socket\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.578825 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.578793    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-workload-certs\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.579587 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.579564    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-token\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.579863 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.579831    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-data\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.580035 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.580010    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-credential-socket\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.580228 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.580211    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istiod-ca-cert\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.581222 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.581200    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istiod-ca-cert\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.582698 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.582120    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-workload-certs\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.582839 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.582344    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-data\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.582946 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.582563    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-credential-socket\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.585266 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.585245    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-podinfo\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.591677 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.591656    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn"
Oct 30 19:01:05.596401 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.596378    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-envoy\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.657084 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.649837    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-token\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.657084 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.653363    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsjnz\" (UniqueName: \"kubernetes.io/projected/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-kube-api-access-wsjnz\") pod \"e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") " pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.707114 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.699920    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:05.805392 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:05.805338    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:06.036981 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:01:06.036949    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d01e582_b7f9_40a0_b166_13474d7f263c.slice/crio-b63762b9a5030ff529ecec24c5f3b664d7635a86900c6094afaa6f879679e2b9 WatchSource:0}: Error finding container b63762b9a5030ff529ecec24c5f3b664d7635a86900c6094afaa6f879679e2b9: Status 404 returned error can't find the container with id b63762b9a5030ff529ecec24c5f3b664d7635a86900c6094afaa6f879679e2b9
Oct 30 19:01:06.038132 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:06.038100    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn"]
Oct 30 19:01:06.052583 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:06.052559    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a4f200e-7gb7p"]
Oct 30 19:01:06.053986 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:06.053916    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn" event={"ID":"4d01e582-b7f9-40a0-b166-13474d7f263c","Type":"ContainerStarted","Data":"b63762b9a5030ff529ecec24c5f3b664d7635a86900c6094afaa6f879679e2b9"}
Oct 30 19:01:06.061705 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:06.061685    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a4f200e-7gb7p"]
Oct 30 19:01:06.066628 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:06.066603    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:06.159140 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:01:06.159104    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a56e5b5_78bf_4b8a_8588_6ede4c1aa62f.slice/crio-bc1390f82ea4a510fee4ac636b96e288482a25141b9380fa5435aa8a87ba3781 WatchSource:0}: Error finding container bc1390f82ea4a510fee4ac636b96e288482a25141b9380fa5435aa8a87ba3781: Status 404 returned error can't find the container with id bc1390f82ea4a510fee4ac636b96e288482a25141b9380fa5435aa8a87ba3781
Oct 30 19:01:06.160811 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:06.160790    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"]
Oct 30 19:01:06.162250 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:06.162211    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:01:06.162351 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:06.162285    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:01:06.162351 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:06.162332    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:01:06.836622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:06.836592    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d4c6b1b-e52c-4b98-bf41-b8de7a282623" path="/var/lib/kubelet/pods/3d4c6b1b-e52c-4b98-bf41-b8de7a282623/volumes"
Oct 30 19:01:07.063504 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:07.063468    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn" event={"ID":"4d01e582-b7f9-40a0-b166-13474d7f263c","Type":"ContainerStarted","Data":"64c93e5d3938ec04fe53faec8a4998d4aaa0966fbcfc3196a81f2b6189343d10"}
Oct 30 19:01:07.063695 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:07.063554    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn"
Oct 30 19:01:07.065216 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:07.065194    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf" event={"ID":"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f","Type":"ContainerStarted","Data":"0aaafcbd95cecf4b847201cfdf0d4c93a0a5f69b040770392c0d9b184a7ce0d9"}
Oct 30 19:01:07.065323 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:07.065232    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf" event={"ID":"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f","Type":"ContainerStarted","Data":"bc1390f82ea4a510fee4ac636b96e288482a25141b9380fa5435aa8a87ba3781"}
Oct 30 19:01:07.175046 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:07.174994    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn" podStartSLOduration=2.174976659 podStartE2EDuration="2.174976659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:07.169867686 +0000 UTC m=+1468.446918130" watchObservedRunningTime="2026-10-30 19:01:07.174976659 +0000 UTC m=+1468.452027103"
Oct 30 19:01:07.284614 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:07.284560    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf" podStartSLOduration=2.284541294 podStartE2EDuration="2.284541294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:07.279549277 +0000 UTC m=+1468.556599726" watchObservedRunningTime="2026-10-30 19:01:07.284541294 +0000 UTC m=+1468.561591740"
Oct 30 19:01:07.700315 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:07.700275    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:07.706498 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:07.706473    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:08.074286 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:08.074191    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:08.075051 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:08.075029    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:01:09.330448 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:09.330414    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:09.522289 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:09.522257    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:09.730888 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:09.730854    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:10.185714 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:10.185673    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:11.249290 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:11.249258    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eb09cf8-mcsfw"]
Oct 30 19:01:11.253610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:11.253594    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb09cf8-mcsfw"
Oct 30 19:01:11.276515 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:11.276492    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef7c3720-9c21-4ba3-b919-44f1009889b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2eb09cf8-mcsfw\" (UID: \"ef7c3720-9c21-4ba3-b919-44f1009889b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb09cf8-mcsfw"
Oct 30 19:01:11.276624 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:11.276607    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef7c3720-9c21-4ba3-b919-44f1009889b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2eb09cf8-mcsfw\" (UID: \"ef7c3720-9c21-4ba3-b919-44f1009889b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb09cf8-mcsfw"
Oct 30 19:01:11.276767 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:11.276753    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh8sq\" (UniqueName: \"kubernetes.io/projected/ef7c3720-9c21-4ba3-b919-44f1009889b4-kube-api-access-jh8sq\") pod \"maas-api-revoke-sub-2eb09cf8-mcsfw\" (UID: \"ef7c3720-9c21-4ba3-b919-44f1009889b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb09cf8-mcsfw"
Oct 30 19:01:11.310809 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:11.310789    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:01:11.357507 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:11.357478    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:01:11.365258 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:11.365224    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eb09cf8-mcsfw"]
Oct 30 19:01:11.371961 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:11.371938    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:01:11.372182 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:11.372164    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:01:11.378311 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:11.378281    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef7c3720-9c21-4ba3-b919-44f1009889b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2eb09cf8-mcsfw\" (UID: \"ef7c3720-9c21-4ba3-b919-44f1009889b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb09cf8-mcsfw"
Oct 30 19:01:11.378430 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:11.378413    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef7c3720-9c21-4ba3-b919-44f1009889b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2eb09cf8-mcsfw\" (UID: \"ef7c3720-9c21-4ba3-b919-44f1009889b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb09cf8-mcsfw"
Oct 30 19:01:11.378589 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:11.378566    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh8sq\" (UniqueName: \"kubernetes.io/projected/ef7c3720-9c21-4ba3-b919-44f1009889b4-kube-api-access-jh8sq\") pod \"maas-api-revoke-sub-2eb09cf8-mcsfw\" (UID: \"ef7c3720-9c21-4ba3-b919-44f1009889b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb09cf8-mcsfw"
Oct 30 19:01:11.379213 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:11.379191    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef7c3720-9c21-4ba3-b919-44f1009889b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2eb09cf8-mcsfw\" (UID: \"ef7c3720-9c21-4ba3-b919-44f1009889b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb09cf8-mcsfw"
Oct 30 19:01:11.381590 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:11.381566    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef7c3720-9c21-4ba3-b919-44f1009889b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2eb09cf8-mcsfw\" (UID: \"ef7c3720-9c21-4ba3-b919-44f1009889b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb09cf8-mcsfw"
Oct 30 19:01:11.439355 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:11.439326    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh8sq\" (UniqueName: \"kubernetes.io/projected/ef7c3720-9c21-4ba3-b919-44f1009889b4-kube-api-access-jh8sq\") pod \"maas-api-revoke-sub-2eb09cf8-mcsfw\" (UID: \"ef7c3720-9c21-4ba3-b919-44f1009889b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb09cf8-mcsfw"
Oct 30 19:01:11.568137 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:11.568041    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb09cf8-mcsfw"
Oct 30 19:01:11.947604 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:01:11.947571    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7c3720_9c21_4ba3_b919_44f1009889b4.slice/crio-9b6f745b4a2984b45d84173d61475966905eb775c742f9ebead1875f9cb447b7 WatchSource:0}: Error finding container 9b6f745b4a2984b45d84173d61475966905eb775c742f9ebead1875f9cb447b7: Status 404 returned error can't find the container with id 9b6f745b4a2984b45d84173d61475966905eb775c742f9ebead1875f9cb447b7
Oct 30 19:01:11.947989 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:11.947967    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eb09cf8-mcsfw"]
Oct 30 19:01:12.109691 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:12.109661    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb09cf8-mcsfw" event={"ID":"ef7c3720-9c21-4ba3-b919-44f1009889b4","Type":"ContainerStarted","Data":"9b6f745b4a2984b45d84173d61475966905eb775c742f9ebead1875f9cb447b7"}
Oct 30 19:01:13.119472 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:13.119443    4924 generic.go:356] "Generic (PLEG): container finished" podID="ef7c3720-9c21-4ba3-b919-44f1009889b4" containerID="e6439e1459a4a5efbb437fd1384f576c3d719f8508898c09d90494f2a12d48a7" exitCode=0
Oct 30 19:01:13.119835 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:13.119526    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb09cf8-mcsfw" event={"ID":"ef7c3720-9c21-4ba3-b919-44f1009889b4","Type":"ContainerDied","Data":"e6439e1459a4a5efbb437fd1384f576c3d719f8508898c09d90494f2a12d48a7"}
Oct 30 19:01:14.285519 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:14.285494    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb09cf8-mcsfw"
Oct 30 19:01:14.321609 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:14.321586    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef7c3720-9c21-4ba3-b919-44f1009889b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef7c3720-9c21-4ba3-b919-44f1009889b4-maas-api-openshift-ca\") pod \"ef7c3720-9c21-4ba3-b919-44f1009889b4\" (UID: \"ef7c3720-9c21-4ba3-b919-44f1009889b4\") "
Oct 30 19:01:14.321723 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:14.321692    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef7c3720-9c21-4ba3-b919-44f1009889b4-kube-api-access-jh8sq\" (UniqueName: \"kubernetes.io/projected/ef7c3720-9c21-4ba3-b919-44f1009889b4-kube-api-access-jh8sq\") pod \"ef7c3720-9c21-4ba3-b919-44f1009889b4\" (UID: \"ef7c3720-9c21-4ba3-b919-44f1009889b4\") "
Oct 30 19:01:14.321885 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:14.321864    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef7c3720-9c21-4ba3-b919-44f1009889b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef7c3720-9c21-4ba3-b919-44f1009889b4-maas-api-xks-ca\") pod \"ef7c3720-9c21-4ba3-b919-44f1009889b4\" (UID: \"ef7c3720-9c21-4ba3-b919-44f1009889b4\") "
Oct 30 19:01:14.322002 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:14.321981    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7c3720-9c21-4ba3-b919-44f1009889b4-maas-api-openshift-ca" pod "ef7c3720-9c21-4ba3-b919-44f1009889b4" (UID: "ef7c3720-9c21-4ba3-b919-44f1009889b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:14.322428 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:14.322411    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef7c3720-9c21-4ba3-b919-44f1009889b4-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:14.323663 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:14.323640    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7c3720-9c21-4ba3-b919-44f1009889b4-kube-api-access-jh8sq" pod "ef7c3720-9c21-4ba3-b919-44f1009889b4" (UID: "ef7c3720-9c21-4ba3-b919-44f1009889b4"). InnerVolumeSpecName "kube-api-access-jh8sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:14.323800 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:14.323782    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7c3720-9c21-4ba3-b919-44f1009889b4-maas-api-xks-ca" pod "ef7c3720-9c21-4ba3-b919-44f1009889b4" (UID: "ef7c3720-9c21-4ba3-b919-44f1009889b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:01:14.423968 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:14.423898    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef7c3720-9c21-4ba3-b919-44f1009889b4-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:14.423968 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:14.423926    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh8sq\" (UniqueName: \"kubernetes.io/projected/ef7c3720-9c21-4ba3-b919-44f1009889b4-kube-api-access-jh8sq\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:14.556731 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:14.556697    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:14.784360 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:14.784268    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:14.952412 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:14.952378    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb"]
Oct 30 19:01:14.953210 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:14.953188    4924 cpu_manager.go:414] "containerMap: removing container" podUID="ef7c3720-9c21-4ba3-b919-44f1009889b4" containerName="revoke-keys"
Oct 30 19:01:14.953210 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:14.953209    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef7c3720-9c21-4ba3-b919-44f1009889b4" containerName="revoke-keys"
Oct 30 19:01:14.953433 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:14.953382    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef7c3720-9c21-4ba3-b919-44f1009889b4" containerName="revoke-keys"
Oct 30 19:01:14.957423 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:14.957400    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb"
Oct 30 19:01:15.009111 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:15.007206    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-85b6ee7b-dockercfg-9jvgr\""
Oct 30 19:01:15.009111 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:15.007223    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-85b6ee7b\""
Oct 30 19:01:15.009849 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:15.009829    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb"]
Oct 30 19:01:15.041241 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:15.041157    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsvdr\" (UniqueName: \"kubernetes.io/projected/d1a29457-36fd-4c0f-b19f-80fab8e0725a-kube-api-access-tsvdr\") pod \"payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb\" (UID: \"d1a29457-36fd-4c0f-b19f-80fab8e0725a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb"
Oct 30 19:01:15.041480 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:15.041457    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1a29457-36fd-4c0f-b19f-80fab8e0725a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb\" (UID: \"d1a29457-36fd-4c0f-b19f-80fab8e0725a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb"
Oct 30 19:01:15.095437 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:15.095405    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m"]
Oct 30 19:01:15.100647 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:15.100610    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m"
Oct 30 19:01:15.137791 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:15.137762    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb09cf8-mcsfw"
Oct 30 19:01:15.137994 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:15.137784    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb09cf8-mcsfw" event={"ID":"ef7c3720-9c21-4ba3-b919-44f1009889b4","Type":"ContainerDied","Data":"9b6f745b4a2984b45d84173d61475966905eb775c742f9ebead1875f9cb447b7"}
Oct 30 19:01:15.137994 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:15.137843    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b6f745b4a2984b45d84173d61475966905eb775c742f9ebead1875f9cb447b7"
Oct 30 19:01:15.142543 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:15.142517    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1f3722a-e299-4178-a91f-1fa91e6014f6-plugins-config-volume\") pod \"payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m\" (UID: \"a1f3722a-e299-4178-a91f-1fa91e6014f6\") " pod="openshift-ingress/payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m"
Oct 30 19:01:15.142705 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:15.142685    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjv5v\" (UniqueName: \"kubernetes.io/projected/a1f3722a-e299-4178-a91f-1fa91e6014f6-kube-api-access-hjv5v\") pod \"payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m\" (UID: \"a1f3722a-e299-4178-a91f-1fa91e6014f6\") " pod="openshift-ingress/payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m"
Oct 30 19:01:15.142836 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:15.142820    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1a29457-36fd-4c0f-b19f-80fab8e0725a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb\" (UID: \"d1a29457-36fd-4c0f-b19f-80fab8e0725a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb"
Oct 30 19:01:15.142957 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:15.142941    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsvdr\" (UniqueName: \"kubernetes.io/projected/d1a29457-36fd-4c0f-b19f-80fab8e0725a-kube-api-access-tsvdr\") pod \"payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb\" (UID: \"d1a29457-36fd-4c0f-b19f-80fab8e0725a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb"
Oct 30 19:01:15.143834 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:15.143804    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1a29457-36fd-4c0f-b19f-80fab8e0725a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb\" (UID: \"d1a29457-36fd-4c0f-b19f-80fab8e0725a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb"
Oct 30 19:01:15.154157 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:15.154131    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m"]
Oct 30 19:01:15.205924 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:15.205852    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsvdr\" (UniqueName: \"kubernetes.io/projected/d1a29457-36fd-4c0f-b19f-80fab8e0725a-kube-api-access-tsvdr\") pod \"payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb\" (UID: \"d1a29457-36fd-4c0f-b19f-80fab8e0725a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb"
Oct 30 19:01:15.244306 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:15.244268    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1f3722a-e299-4178-a91f-1fa91e6014f6-plugins-config-volume\") pod \"payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m\" (UID: \"a1f3722a-e299-4178-a91f-1fa91e6014f6\") " pod="openshift-ingress/payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m"
Oct 30 19:01:15.244480 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:15.244377    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjv5v\" (UniqueName: \"kubernetes.io/projected/a1f3722a-e299-4178-a91f-1fa91e6014f6-kube-api-access-hjv5v\") pod \"payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m\" (UID: \"a1f3722a-e299-4178-a91f-1fa91e6014f6\") " pod="openshift-ingress/payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m"
Oct 30 19:01:15.244948 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:15.244929    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1f3722a-e299-4178-a91f-1fa91e6014f6-plugins-config-volume\") pod \"payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m\" (UID: \"a1f3722a-e299-4178-a91f-1fa91e6014f6\") " pod="openshift-ingress/payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m"
Oct 30 19:01:15.274643 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:15.274618    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb"
Oct 30 19:01:15.308196 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:15.307487    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjv5v\" (UniqueName: \"kubernetes.io/projected/a1f3722a-e299-4178-a91f-1fa91e6014f6-kube-api-access-hjv5v\") pod \"payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m\" (UID: \"a1f3722a-e299-4178-a91f-1fa91e6014f6\") " pod="openshift-ingress/payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m"
Oct 30 19:01:15.417183 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:15.416884    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m"
Oct 30 19:01:15.548036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:15.547968    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:15.715911 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:01:15.715878    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:01:15.743018 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:01:15.742973    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a29457_36fd_4c0f_b19f_80fab8e0725a.slice/crio-29993da22c5d48d2f2e5566f4abeda3771a9b738ff4af00155d0e451ec7c28da WatchSource:0}: Error finding container 29993da22c5d48d2f2e5566f4abeda3771a9b738ff4af00155d0e451ec7c28da: Status 404 returned error can't find the container with id 29993da22c5d48d2f2e5566f4abeda3771a9b738ff4af00155d0e451ec7c28da
Oct 30 19:01:15.744314 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:15.744293    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb"]
Oct 30 19:01:15.890633 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:01:15.890604    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f3722a_e299_4178_a91f_1fa91e6014f6.slice/crio-ed985f74a69eae40e586e3459a9203349d94c5db51f197813c8388338beee6bb WatchSource:0}: Error finding container ed985f74a69eae40e586e3459a9203349d94c5db51f197813c8388338beee6bb: Status 404 returned error can't find the container with id ed985f74a69eae40e586e3459a9203349d94c5db51f197813c8388338beee6bb
Oct 30 19:01:15.892844 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:15.892820    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m"]
Oct 30 19:01:16.148034 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:16.147999    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m" event={"ID":"a1f3722a-e299-4178-a91f-1fa91e6014f6","Type":"ContainerStarted","Data":"b1902f135678ef6c20b33e7feff98eea17dce4bb5089937b44f1bc365127fc59"}
Oct 30 19:01:16.148240 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:16.148075    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m" event={"ID":"a1f3722a-e299-4178-a91f-1fa91e6014f6","Type":"ContainerStarted","Data":"ed985f74a69eae40e586e3459a9203349d94c5db51f197813c8388338beee6bb"}
Oct 30 19:01:16.148240 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:16.148109    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m"
Oct 30 19:01:16.149653 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:16.149629    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb" event={"ID":"d1a29457-36fd-4c0f-b19f-80fab8e0725a","Type":"ContainerStarted","Data":"e09d590b1ed33ecc83314a8363b6b4815af6aa296790d870cfe78c2a4b1c60f3"}
Oct 30 19:01:16.149757 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:16.149670    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb"
Oct 30 19:01:16.149757 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:16.149685    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb" event={"ID":"d1a29457-36fd-4c0f-b19f-80fab8e0725a","Type":"ContainerStarted","Data":"29993da22c5d48d2f2e5566f4abeda3771a9b738ff4af00155d0e451ec7c28da"}
Oct 30 19:01:16.258838 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:16.258790    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m" podStartSLOduration=1.2587773740000001 podStartE2EDuration="1.258777374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:16.257055297 +0000 UTC m=+1477.534105743" watchObservedRunningTime="2026-10-30 19:01:16.258777374 +0000 UTC m=+1477.535827820"
Oct 30 19:01:16.365104 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:16.365027    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb" podStartSLOduration=2.365010263 podStartE2EDuration="2.365010263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:16.364805185 +0000 UTC m=+1477.641855630" watchObservedRunningTime="2026-10-30 19:01:16.365010263 +0000 UTC m=+1477.642060709"
Oct 30 19:01:18.077767 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:18.077738    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn"
Oct 30 19:01:20.797966 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:20.797877    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:21.918796 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:21.918756    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:22.151084 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:22.150514    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:23.540090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:23.537189    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" podUID="eb82dd7d-e682-42a2-924e-a84f21172fc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:23.589091 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:23.588397    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj" podUID="51f3e639-9cd4-4aaa-a568-f19b6d2704fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:23.897089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:23.896137    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5" podUID="925a4de9-b451-4263-9a1c-eebbbae4a08c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:23.897089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:23.896553    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7" podUID="97a3b018-9083-4b7c-af88-6edddc585d67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 19:01:23.929088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:23.915053    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m" podUID="1a90412d-c047-422e-b3c7-9546d286c303" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:23.929088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:23.914990    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2" podUID="fdf4711c-418a-436b-848a-d46974566573" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:23.949178 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:23.949139    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd" podUID="a3d792df-177d-4f7a-bba8-f6351175462e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:23.988203 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:23.988141    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l" podUID="bdc8df4b-8c94-40b8-b4be-80d1432d87de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:23.989474 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:23.989399    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86" podUID="60a58597-fdc4-4722-8d2a-29e475f84a76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 19:01:24.169828 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:24.169372    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld" podUID="f04af09f-b8d2-4b47-b0b3-88eb12adff93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 30 19:01:24.181365 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:24.181225    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:01:24.181365 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:24.181302    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw" podUID="922dfc11-a4ba-4d50-a50f-b5155d277792" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:24.451139 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:24.450150    4924 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt" podUID="955e98b9-864b-4636-9511-8b69f981eaa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:24.493085 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:24.492810    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:24.540093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:24.538200    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" podUID="eb82dd7d-e682-42a2-924e-a84f21172fc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:24.540093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:24.538584    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 19:01:24.550179 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:24.550144    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt" podUID="955e98b9-864b-4636-9511-8b69f981eaa7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:24.621174 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:24.621132    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg" podUID="9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:24.636210 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:24.636175    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj" podUID="51f3e639-9cd4-4aaa-a568-f19b6d2704fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:01:24.637087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:24.636524    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 19:01:24.763140 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:24.755998    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 19:01:24.947224 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:24.943116    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 19:01:25.171087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:25.159382    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:25.396953 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:25.396922    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:26.376336 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.376300    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:26.400956 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.400918    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"]
Oct 30 19:01:26.407974 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.407946    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.463138 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.463102    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"]
Oct 30 19:01:26.504471 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.504436    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7hxw\" (UniqueName: \"kubernetes.io/projected/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-kube-api-access-d7hxw\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.504660 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.504604    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-workload-certs\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.504715 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.504681    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-workload-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.504770 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.504755    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-token\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.504970 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.504952    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istiod-ca-cert\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.505033 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.505018    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-envoy\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.505145 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.505128    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-data\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.505216 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.505161    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-credential-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.505216 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.505195    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-podinfo\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.607240 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.607209    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istiod-ca-cert\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.607409 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.607263    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-envoy\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.607409 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.607289    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-data\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.607409 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.607307    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-credential-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.607409 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.607325    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-podinfo\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.607409 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.607407    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7hxw\" (UniqueName: \"kubernetes.io/projected/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-kube-api-access-d7hxw\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.607708 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.607502    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-workload-certs\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.607708 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.607532    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-workload-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.607708 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.607592    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-token\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.607893 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.607795    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-data\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.607893 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.607812    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-credential-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.608000 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.607922    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-workload-certs\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.608102 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.608079    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-workload-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.608307 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.608289    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istiod-ca-cert\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.609518 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.609500    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-envoy\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.609823 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.609796    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-podinfo\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.663407 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.663346    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-token\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.663711 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.663691    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7hxw\" (UniqueName: \"kubernetes.io/projected/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-kube-api-access-d7hxw\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:26.728870 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:26.728839    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:27.161966 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:27.161662    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb"
Oct 30 19:01:27.162917 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:27.162891    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m"
Oct 30 19:01:27.163885 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:01:27.163858    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa7b7f06_28aa_4a1f_ac1c_d1d363815699.slice/crio-f143be78fd8db90d978dcfaba8963d971ae760ed22e1921e4db78e2085342e1a WatchSource:0}: Error finding container f143be78fd8db90d978dcfaba8963d971ae760ed22e1921e4db78e2085342e1a: Status 404 returned error can't find the container with id f143be78fd8db90d978dcfaba8963d971ae760ed22e1921e4db78e2085342e1a
Oct 30 19:01:27.164122 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:27.164098    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"]
Oct 30 19:01:27.173916 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:27.173873    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:01:27.174180 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:27.174156    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:01:27.174356 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:27.174336    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:01:27.427916 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:27.427812    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2" event={"ID":"fa7b7f06-28aa-4a1f-ac1c-d1d363815699","Type":"ContainerStarted","Data":"30d947932b02122854a92acc189c210cd0e0bdcbf032836c9f5dd7fddcb3afca"}
Oct 30 19:01:27.428400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:27.427917    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2" event={"ID":"fa7b7f06-28aa-4a1f-ac1c-d1d363815699","Type":"ContainerStarted","Data":"f143be78fd8db90d978dcfaba8963d971ae760ed22e1921e4db78e2085342e1a"}
Oct 30 19:01:27.543405 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:27.543348    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2" podStartSLOduration=1.543333975 podStartE2EDuration="1.543333975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:27.537185868 +0000 UTC m=+1488.814236316" watchObservedRunningTime="2026-10-30 19:01:27.543333975 +0000 UTC m=+1488.820384419"
Oct 30 19:01:27.729799 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:27.729713    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:28.731481 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:28.731442    4924 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 19:01:28.731899 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:28.731500    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2" podUID="fa7b7f06-28aa-4a1f-ac1c-d1d363815699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 19:01:28.987908 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:28.987826    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:29.191720 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:29.191677    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:29.483457 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:29.483427    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:29.729589 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:29.729556    4924 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 19:01:29.729757 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:29.729620    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2" podUID="fa7b7f06-28aa-4a1f-ac1c-d1d363815699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 19:01:30.729690 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:30.729634    4924 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 19:01:30.730267 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:30.729693    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2" podUID="fa7b7f06-28aa-4a1f-ac1c-d1d363815699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 19:01:31.646741 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:31.646706    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:31.732089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:31.731598    4924 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 19:01:31.732089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:31.731670    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2" podUID="fa7b7f06-28aa-4a1f-ac1c-d1d363815699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 19:01:32.127089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:32.126892    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:32.336428 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:32.336354    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:32.538051 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:32.537967    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:32.756559 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:32.756517    4924 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 19:01:32.757095 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:32.756582    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2" podUID="fa7b7f06-28aa-4a1f-ac1c-d1d363815699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 19:01:32.828993 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:32.828894    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:33.612657 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:33.612622    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:33.729369 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:33.729335    4924 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 19:01:33.729541 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:33.729382    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2" podUID="fa7b7f06-28aa-4a1f-ac1c-d1d363815699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 19:01:34.734272 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:34.734239    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:34.734707 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:34.734623    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:34.735295 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:34.735278    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:01:36.618310 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:36.618273    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:37.893476 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:37.893442    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:38.437032 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:38.437000    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:39.027369 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:39.027326    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:39.049431 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:39.049399    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 19:01:39.054511 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:39.054493    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 19:01:39.207148 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:39.207120    4924 scope.go:122] "RemoveContainer" containerID="2d9bc7e2db3937e94b4309f2ffc3f3a391af48946d94eadb27681a015533e874"
Oct 30 19:01:39.220967 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:39.220946    4924 scope.go:122] "RemoveContainer" containerID="b900612bb5cfa9e8958e93d14adc5e1657f774872148ea9c719d652c73bdc70c"
Oct 30 19:01:39.230757 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:39.230740    4924 scope.go:122] "RemoveContainer" containerID="07969b7e5541e482f86c43c7ed5f169ebe6811371301f94c608dfe19c3797315"
Oct 30 19:01:39.240110 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:39.240089    4924 scope.go:122] "RemoveContainer" containerID="1081407cf428e398687d8b0fcb479d6ca08567869558c803deb5797f9f2bc315"
Oct 30 19:01:39.248864 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:39.248845    4924 scope.go:122] "RemoveContainer" containerID="480ed17a992e5b1e18ad09d44c2a0deee5d5f0be2d55021ae0c122dd4bb270e0"
Oct 30 19:01:39.257635 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:39.257617    4924 scope.go:122] "RemoveContainer" containerID="a9cfae4340d8a4d1ab301b8d6a3fde9f38fa12e305f944ca27af5901ccde4bae"
Oct 30 19:01:39.267054 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:39.267037    4924 scope.go:122] "RemoveContainer" containerID="eda4d14392e7feefc0e46aeb448447470c07b977d5df707bbe3b44807b4e519a"
Oct 30 19:01:39.276214 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:39.276197    4924 scope.go:122] "RemoveContainer" containerID="451d1a8b0633a125b57a2ff6892faf63089d490b2ea05ebeff8af21e21623264"
Oct 30 19:01:42.111238 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:42.111170    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:44.191108 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:44.191054    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:45.712912 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:01:45.712876    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:01:45.848439 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:45.848406    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:47.146989 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:47.146962    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:47.340535 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:47.340483    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:51.801608 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:51.801569    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:56.553495 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:56.553459    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:56.728991 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:56.728953    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:56.909212 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:56.909183    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:01:58.518620 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:58.518583    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e07bd9b4-k4rk5"]
Oct 30 19:01:58.522985 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:58.522968    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07bd9b4-k4rk5"
Oct 30 19:01:58.542961 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:58.542933    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9828d1fc-c17d-4cd4-947f-3ef34336be4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e07bd9b4-k4rk5\" (UID: \"9828d1fc-c17d-4cd4-947f-3ef34336be4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07bd9b4-k4rk5"
Oct 30 19:01:58.543137 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:58.542969    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9828d1fc-c17d-4cd4-947f-3ef34336be4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e07bd9b4-k4rk5\" (UID: \"9828d1fc-c17d-4cd4-947f-3ef34336be4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07bd9b4-k4rk5"
Oct 30 19:01:58.543214 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:58.543175    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7mk2\" (UniqueName: \"kubernetes.io/projected/9828d1fc-c17d-4cd4-947f-3ef34336be4a-kube-api-access-h7mk2\") pod \"maas-api-revoke-sub-e07bd9b4-k4rk5\" (UID: \"9828d1fc-c17d-4cd4-947f-3ef34336be4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07bd9b4-k4rk5"
Oct 30 19:01:58.578767 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:58.578732    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e07bd9b4-k4rk5"]
Oct 30 19:01:58.618994 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:58.618963    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:01:58.619266 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:58.618963    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:01:58.619365 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:58.618975    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:01:58.619365 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:58.619100    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:01:58.644439 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:58.644410    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9828d1fc-c17d-4cd4-947f-3ef34336be4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e07bd9b4-k4rk5\" (UID: \"9828d1fc-c17d-4cd4-947f-3ef34336be4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07bd9b4-k4rk5"
Oct 30 19:01:58.644598 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:58.644446    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9828d1fc-c17d-4cd4-947f-3ef34336be4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e07bd9b4-k4rk5\" (UID: \"9828d1fc-c17d-4cd4-947f-3ef34336be4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07bd9b4-k4rk5"
Oct 30 19:01:58.644598 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:58.644519    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7mk2\" (UniqueName: \"kubernetes.io/projected/9828d1fc-c17d-4cd4-947f-3ef34336be4a-kube-api-access-h7mk2\") pod \"maas-api-revoke-sub-e07bd9b4-k4rk5\" (UID: \"9828d1fc-c17d-4cd4-947f-3ef34336be4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07bd9b4-k4rk5"
Oct 30 19:01:58.645270 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:58.645249    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9828d1fc-c17d-4cd4-947f-3ef34336be4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e07bd9b4-k4rk5\" (UID: \"9828d1fc-c17d-4cd4-947f-3ef34336be4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07bd9b4-k4rk5"
Oct 30 19:01:58.646849 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:58.646829    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9828d1fc-c17d-4cd4-947f-3ef34336be4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e07bd9b4-k4rk5\" (UID: \"9828d1fc-c17d-4cd4-947f-3ef34336be4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07bd9b4-k4rk5"
Oct 30 19:01:58.698749 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:58.698712    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7mk2\" (UniqueName: \"kubernetes.io/projected/9828d1fc-c17d-4cd4-947f-3ef34336be4a-kube-api-access-h7mk2\") pod \"maas-api-revoke-sub-e07bd9b4-k4rk5\" (UID: \"9828d1fc-c17d-4cd4-947f-3ef34336be4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07bd9b4-k4rk5"
Oct 30 19:01:58.836538 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:58.836462    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07bd9b4-k4rk5"
Oct 30 19:01:59.221688 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:01:59.221658    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9828d1fc_c17d_4cd4_947f_3ef34336be4a.slice/crio-ba9b2468652f7f01ce4bf1ce5d160a0f3c2bd3bdba191a03ead6f0d666684e01 WatchSource:0}: Error finding container ba9b2468652f7f01ce4bf1ce5d160a0f3c2bd3bdba191a03ead6f0d666684e01: Status 404 returned error can't find the container with id ba9b2468652f7f01ce4bf1ce5d160a0f3c2bd3bdba191a03ead6f0d666684e01
Oct 30 19:01:59.222560 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:59.222537    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e07bd9b4-k4rk5"]
Oct 30 19:01:59.822312 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:59.822195    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07bd9b4-k4rk5" event={"ID":"9828d1fc-c17d-4cd4-947f-3ef34336be4a","Type":"ContainerStarted","Data":"7d0684ac3f957a1089b3acccc15133ae523469cb641bfc1dfaab2c3d3506adaf"}
Oct 30 19:01:59.822312 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:59.822272    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07bd9b4-k4rk5" event={"ID":"9828d1fc-c17d-4cd4-947f-3ef34336be4a","Type":"ContainerStarted","Data":"ba9b2468652f7f01ce4bf1ce5d160a0f3c2bd3bdba191a03ead6f0d666684e01"}
Oct 30 19:01:59.933233 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:01:59.933184    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07bd9b4-k4rk5" podStartSLOduration=1.933169369 podStartE2EDuration="1.933169369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:59.930516978 +0000 UTC m=+1521.207567423" watchObservedRunningTime="2026-10-30 19:01:59.933169369 +0000 UTC m=+1521.210219836"
Oct 30 19:02:00.833867 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:00.833833    4924 generic.go:356] "Generic (PLEG): container finished" podID="9828d1fc-c17d-4cd4-947f-3ef34336be4a" containerID="7d0684ac3f957a1089b3acccc15133ae523469cb641bfc1dfaab2c3d3506adaf" exitCode=0
Oct 30 19:02:00.837930 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:00.837899    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07bd9b4-k4rk5" event={"ID":"9828d1fc-c17d-4cd4-947f-3ef34336be4a","Type":"ContainerDied","Data":"7d0684ac3f957a1089b3acccc15133ae523469cb641bfc1dfaab2c3d3506adaf"}
Oct 30 19:02:01.622782 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:01.622745    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:02.027600 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:02.027570    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07bd9b4-k4rk5"
Oct 30 19:02:02.097468 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:02.097433    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9828d1fc-c17d-4cd4-947f-3ef34336be4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9828d1fc-c17d-4cd4-947f-3ef34336be4a-maas-api-openshift-ca\") pod \"9828d1fc-c17d-4cd4-947f-3ef34336be4a\" (UID: \"9828d1fc-c17d-4cd4-947f-3ef34336be4a\") "
Oct 30 19:02:02.097640 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:02.097623    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9828d1fc-c17d-4cd4-947f-3ef34336be4a-kube-api-access-h7mk2\" (UniqueName: \"kubernetes.io/projected/9828d1fc-c17d-4cd4-947f-3ef34336be4a-kube-api-access-h7mk2\") pod \"9828d1fc-c17d-4cd4-947f-3ef34336be4a\" (UID: \"9828d1fc-c17d-4cd4-947f-3ef34336be4a\") "
Oct 30 19:02:02.097809 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:02.097786    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9828d1fc-c17d-4cd4-947f-3ef34336be4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9828d1fc-c17d-4cd4-947f-3ef34336be4a-maas-api-xks-ca\") pod \"9828d1fc-c17d-4cd4-947f-3ef34336be4a\" (UID: \"9828d1fc-c17d-4cd4-947f-3ef34336be4a\") "
Oct 30 19:02:02.097861 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:02.097840    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9828d1fc-c17d-4cd4-947f-3ef34336be4a-maas-api-openshift-ca" pod "9828d1fc-c17d-4cd4-947f-3ef34336be4a" (UID: "9828d1fc-c17d-4cd4-947f-3ef34336be4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:02:02.098404 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:02.098382    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9828d1fc-c17d-4cd4-947f-3ef34336be4a-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:02.099696 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:02.099665    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9828d1fc-c17d-4cd4-947f-3ef34336be4a-maas-api-xks-ca" pod "9828d1fc-c17d-4cd4-947f-3ef34336be4a" (UID: "9828d1fc-c17d-4cd4-947f-3ef34336be4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:02:02.099795 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:02.099780    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9828d1fc-c17d-4cd4-947f-3ef34336be4a-kube-api-access-h7mk2" pod "9828d1fc-c17d-4cd4-947f-3ef34336be4a" (UID: "9828d1fc-c17d-4cd4-947f-3ef34336be4a"). InnerVolumeSpecName "kube-api-access-h7mk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:02:02.200113 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:02.200016    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7mk2\" (UniqueName: \"kubernetes.io/projected/9828d1fc-c17d-4cd4-947f-3ef34336be4a-kube-api-access-h7mk2\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:02.200113 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:02.200051    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9828d1fc-c17d-4cd4-947f-3ef34336be4a-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:02.853740 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:02.853711    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07bd9b4-k4rk5"
Oct 30 19:02:02.853912 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:02.853739    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07bd9b4-k4rk5" event={"ID":"9828d1fc-c17d-4cd4-947f-3ef34336be4a","Type":"ContainerDied","Data":"ba9b2468652f7f01ce4bf1ce5d160a0f3c2bd3bdba191a03ead6f0d666684e01"}
Oct 30 19:02:02.853912 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:02.853793    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba9b2468652f7f01ce4bf1ce5d160a0f3c2bd3bdba191a03ead6f0d666684e01"
Oct 30 19:02:05.779510 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.779477    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"]
Oct 30 19:02:05.780258 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.780240    4924 cpu_manager.go:414] "containerMap: removing container" podUID="9828d1fc-c17d-4cd4-947f-3ef34336be4a" containerName="revoke-keys"
Oct 30 19:02:05.780258 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.780256    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9828d1fc-c17d-4cd4-947f-3ef34336be4a" containerName="revoke-keys"
Oct 30 19:02:05.780452 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.780373    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="9828d1fc-c17d-4cd4-947f-3ef34336be4a" containerName="revoke-keys"
Oct 30 19:02:05.784335 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.784315    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"
Oct 30 19:02:05.838121 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.838091    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"]
Oct 30 19:02:05.853948 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.852944    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-home\") pod \"distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb\" (UID: \"3cc610a7-5f70-487c-ba64-452ee8458442\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"
Oct 30 19:02:05.854122 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.854038    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2btxd\" (UniqueName: \"kubernetes.io/projected/3cc610a7-5f70-487c-ba64-452ee8458442-kube-api-access-2btxd\") pod \"distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb\" (UID: \"3cc610a7-5f70-487c-ba64-452ee8458442\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"
Oct 30 19:02:05.854300 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.854280    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cc610a7-5f70-487c-ba64-452ee8458442-tls-certs\") pod \"distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb\" (UID: \"3cc610a7-5f70-487c-ba64-452ee8458442\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"
Oct 30 19:02:05.854373 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.854321    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-dshm\") pod \"distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb\" (UID: \"3cc610a7-5f70-487c-ba64-452ee8458442\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"
Oct 30 19:02:05.854373 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.854367    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-tmp-dir\") pod \"distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb\" (UID: \"3cc610a7-5f70-487c-ba64-452ee8458442\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"
Oct 30 19:02:05.854622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.854598    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-model-cache\") pod \"distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb\" (UID: \"3cc610a7-5f70-487c-ba64-452ee8458442\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"
Oct 30 19:02:05.878880 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.878857    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-649d16\"/\"openshift-service-ca.crt\""
Oct 30 19:02:05.879156 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.878896    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-649d16\"/\"kube-root-ca.crt\""
Oct 30 19:02:05.881361 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.881079    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-649d16\"/\"default-dockercfg-m5khk\""
Oct 30 19:02:05.881361 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.881144    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-649d16\"/\"distinct-2-w6-649d16-kserve-self-signed-certs\""
Oct 30 19:02:05.920128 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.920100    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:05.958144 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.956172    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-model-cache\") pod \"distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb\" (UID: \"3cc610a7-5f70-487c-ba64-452ee8458442\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"
Oct 30 19:02:05.958144 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.956318    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-home\") pod \"distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb\" (UID: \"3cc610a7-5f70-487c-ba64-452ee8458442\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"
Oct 30 19:02:05.958144 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.956425    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2btxd\" (UniqueName: \"kubernetes.io/projected/3cc610a7-5f70-487c-ba64-452ee8458442-kube-api-access-2btxd\") pod \"distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb\" (UID: \"3cc610a7-5f70-487c-ba64-452ee8458442\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"
Oct 30 19:02:05.958144 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.956519    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cc610a7-5f70-487c-ba64-452ee8458442-tls-certs\") pod \"distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb\" (UID: \"3cc610a7-5f70-487c-ba64-452ee8458442\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"
Oct 30 19:02:05.958144 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.956575    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-dshm\") pod \"distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb\" (UID: \"3cc610a7-5f70-487c-ba64-452ee8458442\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"
Oct 30 19:02:05.958144 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.956621    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-tmp-dir\") pod \"distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb\" (UID: \"3cc610a7-5f70-487c-ba64-452ee8458442\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"
Oct 30 19:02:05.958144 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.957601    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-model-cache\") pod \"distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb\" (UID: \"3cc610a7-5f70-487c-ba64-452ee8458442\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"
Oct 30 19:02:05.958144 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.957892    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-home\") pod \"distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb\" (UID: \"3cc610a7-5f70-487c-ba64-452ee8458442\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"
Oct 30 19:02:05.959838 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.959806    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-tmp-dir\") pod \"distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb\" (UID: \"3cc610a7-5f70-487c-ba64-452ee8458442\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"
Oct 30 19:02:05.961785 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.961761    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cc610a7-5f70-487c-ba64-452ee8458442-tls-certs\") pod \"distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb\" (UID: \"3cc610a7-5f70-487c-ba64-452ee8458442\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"
Oct 30 19:02:05.962040 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:05.962021    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-dshm\") pod \"distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb\" (UID: \"3cc610a7-5f70-487c-ba64-452ee8458442\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"
Oct 30 19:02:06.012219 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:06.012186    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2btxd\" (UniqueName: \"kubernetes.io/projected/3cc610a7-5f70-487c-ba64-452ee8458442-kube-api-access-2btxd\") pod \"distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb\" (UID: \"3cc610a7-5f70-487c-ba64-452ee8458442\") " pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"
Oct 30 19:02:06.097949 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:06.097924    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"
Oct 30 19:02:06.388959 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:06.382341    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:06.472028 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:02:06.471992    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3cc610a7_5f70_487c_ba64_452ee8458442.slice/crio-603890db6f4bf913447366612aff886fdfbf0e7be0de7adaf525617b29d73cc4 WatchSource:0}: Error finding container 603890db6f4bf913447366612aff886fdfbf0e7be0de7adaf525617b29d73cc4: Status 404 returned error can't find the container with id 603890db6f4bf913447366612aff886fdfbf0e7be0de7adaf525617b29d73cc4
Oct 30 19:02:06.472651 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:06.472626    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"]
Oct 30 19:02:06.897404 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:06.897369    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb" event={"ID":"3cc610a7-5f70-487c-ba64-452ee8458442","Type":"ContainerStarted","Data":"8fd8a489323b20e17dab47bfcb67a0b224a1cdcc33723a5e743381e23eb52ea5"}
Oct 30 19:02:06.897902 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:06.897416    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb" event={"ID":"3cc610a7-5f70-487c-ba64-452ee8458442","Type":"ContainerStarted","Data":"603890db6f4bf913447366612aff886fdfbf0e7be0de7adaf525617b29d73cc4"}
Oct 30 19:02:07.007608 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:07.007556    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb" podStartSLOduration=2.007543508 podStartE2EDuration="2.007543508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:07.005183037 +0000 UTC m=+1528.282233483" watchObservedRunningTime="2026-10-30 19:02:07.007543508 +0000 UTC m=+1528.284593953"
Oct 30 19:02:07.098389 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:07.098361    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"
Oct 30 19:02:09.221858 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:09.221822    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb8901fc-z8s85"]
Oct 30 19:02:09.227908 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:09.227890    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8901fc-z8s85"
Oct 30 19:02:09.280394 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:09.280358    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb8901fc-z8s85"]
Oct 30 19:02:09.306469 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:09.306442    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:02:09.306469 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:09.306447    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:02:09.306716 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:09.306473    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:02:09.306978 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:09.306949    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlkxw\" (UniqueName: \"kubernetes.io/projected/ad056395-5148-4f95-97cc-69163d684a81-kube-api-access-rlkxw\") pod \"maas-api-revoke-sub-bb8901fc-z8s85\" (UID: \"ad056395-5148-4f95-97cc-69163d684a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8901fc-z8s85"
Oct 30 19:02:09.307216 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:09.307192    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad056395-5148-4f95-97cc-69163d684a81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb8901fc-z8s85\" (UID: \"ad056395-5148-4f95-97cc-69163d684a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8901fc-z8s85"
Oct 30 19:02:09.307384 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:09.307362    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad056395-5148-4f95-97cc-69163d684a81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb8901fc-z8s85\" (UID: \"ad056395-5148-4f95-97cc-69163d684a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8901fc-z8s85"
Oct 30 19:02:09.320280 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:09.320249    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:02:09.408856 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:09.408821    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlkxw\" (UniqueName: \"kubernetes.io/projected/ad056395-5148-4f95-97cc-69163d684a81-kube-api-access-rlkxw\") pod \"maas-api-revoke-sub-bb8901fc-z8s85\" (UID: \"ad056395-5148-4f95-97cc-69163d684a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8901fc-z8s85"
Oct 30 19:02:09.409030 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:09.408943    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad056395-5148-4f95-97cc-69163d684a81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb8901fc-z8s85\" (UID: \"ad056395-5148-4f95-97cc-69163d684a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8901fc-z8s85"
Oct 30 19:02:09.409030 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:09.409008    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad056395-5148-4f95-97cc-69163d684a81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb8901fc-z8s85\" (UID: \"ad056395-5148-4f95-97cc-69163d684a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8901fc-z8s85"
Oct 30 19:02:09.409638 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:09.409614    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad056395-5148-4f95-97cc-69163d684a81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb8901fc-z8s85\" (UID: \"ad056395-5148-4f95-97cc-69163d684a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8901fc-z8s85"
Oct 30 19:02:09.411335 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:09.411314    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad056395-5148-4f95-97cc-69163d684a81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb8901fc-z8s85\" (UID: \"ad056395-5148-4f95-97cc-69163d684a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8901fc-z8s85"
Oct 30 19:02:09.462820 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:09.462789    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlkxw\" (UniqueName: \"kubernetes.io/projected/ad056395-5148-4f95-97cc-69163d684a81-kube-api-access-rlkxw\") pod \"maas-api-revoke-sub-bb8901fc-z8s85\" (UID: \"ad056395-5148-4f95-97cc-69163d684a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8901fc-z8s85"
Oct 30 19:02:09.542938 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:09.542856    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8901fc-z8s85"
Oct 30 19:02:09.920898 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:09.920865    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb8901fc-z8s85"]
Oct 30 19:02:10.933565 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:10.933528    4924 generic.go:356] "Generic (PLEG): container finished" podID="ad056395-5148-4f95-97cc-69163d684a81" containerID="c409b0878e573af81af78dd42cb916eb28793520dff805bbf6cfb8981132398c" exitCode=0
Oct 30 19:02:10.934010 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:10.933627    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8901fc-z8s85" event={"ID":"ad056395-5148-4f95-97cc-69163d684a81","Type":"ContainerDied","Data":"c409b0878e573af81af78dd42cb916eb28793520dff805bbf6cfb8981132398c"}
Oct 30 19:02:10.934010 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:10.933740    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8901fc-z8s85" event={"ID":"ad056395-5148-4f95-97cc-69163d684a81","Type":"ContainerStarted","Data":"7c13097b06d982f24f37cb2c579ca87a920dc9845cec48f7b49c1a4439bf94b0"}
Oct 30 19:02:11.220316 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:11.220278    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:12.265684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:12.265654    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8901fc-z8s85"
Oct 30 19:02:12.378115 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:12.372698    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad056395-5148-4f95-97cc-69163d684a81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad056395-5148-4f95-97cc-69163d684a81-maas-api-openshift-ca\") pod \"ad056395-5148-4f95-97cc-69163d684a81\" (UID: \"ad056395-5148-4f95-97cc-69163d684a81\") "
Oct 30 19:02:12.378115 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:12.372814    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad056395-5148-4f95-97cc-69163d684a81-kube-api-access-rlkxw\" (UniqueName: \"kubernetes.io/projected/ad056395-5148-4f95-97cc-69163d684a81-kube-api-access-rlkxw\") pod \"ad056395-5148-4f95-97cc-69163d684a81\" (UID: \"ad056395-5148-4f95-97cc-69163d684a81\") "
Oct 30 19:02:12.378115 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:12.372890    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad056395-5148-4f95-97cc-69163d684a81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad056395-5148-4f95-97cc-69163d684a81-maas-api-xks-ca\") pod \"ad056395-5148-4f95-97cc-69163d684a81\" (UID: \"ad056395-5148-4f95-97cc-69163d684a81\") "
Oct 30 19:02:12.379321 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:12.379233    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad056395-5148-4f95-97cc-69163d684a81-maas-api-openshift-ca" pod "ad056395-5148-4f95-97cc-69163d684a81" (UID: "ad056395-5148-4f95-97cc-69163d684a81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:02:12.396098 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:12.393877    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad056395-5148-4f95-97cc-69163d684a81-maas-api-xks-ca" pod "ad056395-5148-4f95-97cc-69163d684a81" (UID: "ad056395-5148-4f95-97cc-69163d684a81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:02:12.399555 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:12.398824    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad056395-5148-4f95-97cc-69163d684a81-kube-api-access-rlkxw" pod "ad056395-5148-4f95-97cc-69163d684a81" (UID: "ad056395-5148-4f95-97cc-69163d684a81"). InnerVolumeSpecName "kube-api-access-rlkxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:02:12.504523 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:12.504435    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad056395-5148-4f95-97cc-69163d684a81-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:12.504771 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:12.504745    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlkxw\" (UniqueName: \"kubernetes.io/projected/ad056395-5148-4f95-97cc-69163d684a81-kube-api-access-rlkxw\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:12.504887 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:12.504873    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad056395-5148-4f95-97cc-69163d684a81-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:12.954449 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:12.954420    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8901fc-z8s85"
Oct 30 19:02:12.954621 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:12.954451    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb8901fc-z8s85" event={"ID":"ad056395-5148-4f95-97cc-69163d684a81","Type":"ContainerDied","Data":"7c13097b06d982f24f37cb2c579ca87a920dc9845cec48f7b49c1a4439bf94b0"}
Oct 30 19:02:12.954621 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:12.954507    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c13097b06d982f24f37cb2c579ca87a920dc9845cec48f7b49c1a4439bf94b0"
Oct 30 19:02:15.763148 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:02:15.763115    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:02:16.072502 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:16.068681    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:16.098834 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:16.098768    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"
Oct 30 19:02:16.165592 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:16.165561    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"
Oct 30 19:02:17.068571 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:17.068547    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"
Oct 30 19:02:20.662176 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:20.662137    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:20.862681 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:20.862633    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:21.280660 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:21.280617    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:22.145318 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:22.145279    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:22.726595 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:22.726564    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"]
Oct 30 19:02:22.757956 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:22.727828    4924 cpu_manager.go:414] "containerMap: removing container" podUID="ad056395-5148-4f95-97cc-69163d684a81" containerName="revoke-keys"
Oct 30 19:02:22.757956 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:22.727851    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad056395-5148-4f95-97cc-69163d684a81" containerName="revoke-keys"
Oct 30 19:02:22.757956 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:22.728043    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad056395-5148-4f95-97cc-69163d684a81" containerName="revoke-keys"
Oct 30 19:02:22.766427 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:22.764448    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:22.823392 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:22.823356    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"]
Oct 30 19:02:22.836434 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:22.836321    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:22.865476 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:22.865440    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-06203ab6-openshift-default-dockercfg-2vb7x\""
Oct 30 19:02:22.938909 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:22.938871    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fstj2\" (UniqueName: \"kubernetes.io/projected/61957487-4e2c-47d7-9977-1d34f3c2534f-kube-api-access-fstj2\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:22.939137 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:22.939008    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-data\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:22.939216 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:22.939139    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-workload-socket\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:22.939277 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:22.939257    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-envoy\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:22.939332 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:22.939315    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61957487-4e2c-47d7-9977-1d34f3c2534f-istiod-ca-cert\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:22.939388 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:22.939347    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-token\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:22.939507 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:22.939473    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-credential-socket\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:22.939588 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:22.939574    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-workload-certs\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:22.939721 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:22.939701    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-podinfo\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:23.041319 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:23.041228    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-data\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:23.041319 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:23.041288    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-workload-socket\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:23.041672 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:23.041328    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-envoy\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:23.041672 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:23.041356    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61957487-4e2c-47d7-9977-1d34f3c2534f-istiod-ca-cert\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:23.041672 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:23.041392    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-token\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:23.041672 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:23.041443    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-credential-socket\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:23.041672 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:23.041502    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-workload-certs\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:23.041672 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:23.041589    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-podinfo\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:23.041672 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:23.041645    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fstj2\" (UniqueName: \"kubernetes.io/projected/61957487-4e2c-47d7-9977-1d34f3c2534f-kube-api-access-fstj2\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:23.042087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:23.041833    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-data\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:23.043151 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:23.042131    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-credential-socket\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:23.043151 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:23.042180    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-workload-certs\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:23.043151 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:23.042257    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-workload-socket\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:23.043151 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:23.042488    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61957487-4e2c-47d7-9977-1d34f3c2534f-istiod-ca-cert\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:23.044091 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:23.044052    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-envoy\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:23.044364 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:23.044346    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-podinfo\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:23.098003 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:23.097972    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-token\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:23.098259 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:23.098235    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fstj2\" (UniqueName: \"kubernetes.io/projected/61957487-4e2c-47d7-9977-1d34f3c2534f-kube-api-access-fstj2\") pod \"e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") " pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:23.140436 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:23.140399    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:23.202523 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:23.202482    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:23.669791 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:23.607141    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"]
Oct 30 19:02:23.669791 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:02:23.660618    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61957487_4e2c_47d7_9977_1d34f3c2534f.slice/crio-c51e01b837132b568d43541aa4b8c38e9b4d76daf4798ad2157b73e5293ffd7d WatchSource:0}: Error finding container c51e01b837132b568d43541aa4b8c38e9b4d76daf4798ad2157b73e5293ffd7d: Status 404 returned error can't find the container with id c51e01b837132b568d43541aa4b8c38e9b4d76daf4798ad2157b73e5293ffd7d
Oct 30 19:02:23.669791 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:23.663828    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:02:23.669791 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:23.663944    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:02:23.669791 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:23.663984    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:02:23.911901 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:23.911865    4924 patch_prober.go:29] interesting pod/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": read tcp 10.132.0.2:40390->10.132.0.139:15021: read: connection reset by peer" start-of-body=
Oct 30 19:02:23.912026 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:23.911919    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf" podUID="4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": read tcp 10.132.0.2:40390->10.132.0.139:15021: read: connection reset by peer"
Oct 30 19:02:24.127455 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:24.127423    4924 generic.go:356] "Generic (PLEG): container finished" podID="4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f" containerID="0aaafcbd95cecf4b847201cfdf0d4c93a0a5f69b040770392c0d9b184a7ce0d9" exitCode=0
Oct 30 19:02:24.127653 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:24.127495    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf" event={"ID":"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f","Type":"ContainerDied","Data":"0aaafcbd95cecf4b847201cfdf0d4c93a0a5f69b040770392c0d9b184a7ce0d9"}
Oct 30 19:02:24.128108 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:24.128045    4924 scope.go:122] "RemoveContainer" containerID="0aaafcbd95cecf4b847201cfdf0d4c93a0a5f69b040770392c0d9b184a7ce0d9"
Oct 30 19:02:24.129557 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:24.129536    4924 generic.go:356] "Generic (PLEG): container finished" podID="82102463-ed46-40e7-8c8f-becb671573e7" containerID="96c968c38085700c1c8e86121b29f7e5509fb9650ec2cebcac23adfd3c5d680c" exitCode=0
Oct 30 19:02:24.129666 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:24.129619    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v" event={"ID":"82102463-ed46-40e7-8c8f-becb671573e7","Type":"ContainerDied","Data":"96c968c38085700c1c8e86121b29f7e5509fb9650ec2cebcac23adfd3c5d680c"}
Oct 30 19:02:24.129724 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:24.129681    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:02:24.129787 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:24.129767    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:02:24.129852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:24.129824    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:02:24.130079 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:24.130043    4924 scope.go:122] "RemoveContainer" containerID="96c968c38085700c1c8e86121b29f7e5509fb9650ec2cebcac23adfd3c5d680c"
Oct 30 19:02:24.131717 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:24.131678    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:02:24.131795 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:24.131773    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:02:24.131839 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:24.131825    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:02:24.132433 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:24.132356    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9" event={"ID":"61957487-4e2c-47d7-9977-1d34f3c2534f","Type":"ContainerStarted","Data":"24cf43daf4b82557e9a952375329352223416944d7697fc9e9ea03026a2c27ea"}
Oct 30 19:02:24.132433 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:24.132402    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9" event={"ID":"61957487-4e2c-47d7-9977-1d34f3c2534f","Type":"ContainerStarted","Data":"c51e01b837132b568d43541aa4b8c38e9b4d76daf4798ad2157b73e5293ffd7d"}
Oct 30 19:02:24.141009 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:24.140987    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:24.143256 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:24.143231    4924 patch_prober.go:29] interesting pod/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:02:24.143337 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:24.143278    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9" podUID="61957487-4e2c-47d7-9977-1d34f3c2534f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 19:02:24.467928 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:24.467818    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9" podStartSLOduration=2.467796397 podStartE2EDuration="2.467796397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:24.466633137 +0000 UTC m=+1545.743683583" watchObservedRunningTime="2026-10-30 19:02:24.467796397 +0000 UTC m=+1545.744846844"
Oct 30 19:02:25.158252 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:25.158219    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:25.159492 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:25.159457    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf" event={"ID":"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f","Type":"ContainerStarted","Data":"8897ef6c9480c61746d8647531758504ea98ac0120d0e3b40e6fcc8b0d87c553"}
Oct 30 19:02:25.161719 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:25.161691    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v" event={"ID":"82102463-ed46-40e7-8c8f-becb671573e7","Type":"ContainerStarted","Data":"2747e0cc030a9e55545cbdec7f5567b0fdca65aa57fd13f1e26c8c9f0df469af"}
Oct 30 19:02:25.162180 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:25.162160    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:25.162847 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:25.162830    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:02:25.700959 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:25.700929    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:02:25.701404 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:25.701080    4924 patch_prober.go:29] interesting pod/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:02:25.701404 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:25.701122    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf" podUID="4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 19:02:25.864488 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:25.864457    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:02:25.864695 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:25.864550    4924 patch_prober.go:29] interesting pod/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 19:02:25.864695 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:25.864597    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v" podUID="82102463-ed46-40e7-8c8f-becb671573e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 19:02:25.939093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:25.928918    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:26.700612 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:26.700282    4924 patch_prober.go:29] interesting pod/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:02:26.700612 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:26.700330    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf" podUID="4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 19:02:26.865459 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:26.865430    4924 patch_prober.go:29] interesting pod/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 19:02:26.865895 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:26.865874    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v" podUID="82102463-ed46-40e7-8c8f-becb671573e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 19:02:27.394820 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:27.394793    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:27.623090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:27.619978    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:27.700937 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:27.700857    4924 patch_prober.go:29] interesting pod/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:02:27.700937 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:27.700908    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf" podUID="4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 19:02:27.864907 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:27.864873    4924 patch_prober.go:29] interesting pod/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 19:02:27.865099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:27.864920    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v" podUID="82102463-ed46-40e7-8c8f-becb671573e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 19:02:28.700623 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:28.700586    4924 patch_prober.go:29] interesting pod/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:02:28.701042 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:28.700644    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf" podUID="4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 19:02:28.865409 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:28.865372    4924 patch_prober.go:29] interesting pod/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 19:02:28.865583 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:28.865439    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v" podUID="82102463-ed46-40e7-8c8f-becb671573e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 19:02:29.700973 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:29.700938    4924 patch_prober.go:29] interesting pod/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:02:29.701413 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:29.700989    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf" podUID="4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 19:02:29.864623 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:29.864589    4924 patch_prober.go:29] interesting pod/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 30 19:02:29.864890 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:29.864866    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v" podUID="82102463-ed46-40e7-8c8f-becb671573e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 30 19:02:30.384973 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:30.384932    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:30.570087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:30.565047    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:30.708835 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:30.708736    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:02:30.710500 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:30.710477    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:02:30.711056 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:30.711028    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:02:30.743594 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:30.743558    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9"]
Oct 30 19:02:30.749458 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:30.749419    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9"
Oct 30 19:02:30.770987 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:30.770956    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m45b\" (UniqueName: \"kubernetes.io/projected/02fb175b-4f55-480a-a3d0-6ea00d85adaa-kube-api-access-8m45b\") pod \"payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9\" (UID: \"02fb175b-4f55-480a-a3d0-6ea00d85adaa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9"
Oct 30 19:02:30.771207 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:30.771183    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fb175b-4f55-480a-a3d0-6ea00d85adaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9\" (UID: \"02fb175b-4f55-480a-a3d0-6ea00d85adaa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9"
Oct 30 19:02:30.802837 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:30.802803    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9"]
Oct 30 19:02:30.806356 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:30.806320    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-06203ab6-dockercfg-lrcsb\""
Oct 30 19:02:30.880600 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:30.880555    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fb175b-4f55-480a-a3d0-6ea00d85adaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9\" (UID: \"02fb175b-4f55-480a-a3d0-6ea00d85adaa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9"
Oct 30 19:02:30.880838 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:30.880793    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m45b\" (UniqueName: \"kubernetes.io/projected/02fb175b-4f55-480a-a3d0-6ea00d85adaa-kube-api-access-8m45b\") pod \"payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9\" (UID: \"02fb175b-4f55-480a-a3d0-6ea00d85adaa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9"
Oct 30 19:02:30.886107 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:30.886075    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-06203ab6\""
Oct 30 19:02:30.888222 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:30.888197    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:02:30.888827 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:30.888808    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:02:30.889681 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:30.889664    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:02:30.892750 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:30.892723    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fb175b-4f55-480a-a3d0-6ea00d85adaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9\" (UID: \"02fb175b-4f55-480a-a3d0-6ea00d85adaa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9"
Oct 30 19:02:30.925401 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:30.925352    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c"]
Oct 30 19:02:30.931356 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:30.931329    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c"
Oct 30 19:02:30.953129 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:30.944628    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m45b\" (UniqueName: \"kubernetes.io/projected/02fb175b-4f55-480a-a3d0-6ea00d85adaa-kube-api-access-8m45b\") pod \"payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9\" (UID: \"02fb175b-4f55-480a-a3d0-6ea00d85adaa\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9"
Oct 30 19:02:30.980219 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:30.980189    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c"]
Oct 30 19:02:31.077605 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:31.071355    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9"
Oct 30 19:02:31.086589 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:31.085738    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwxcg\" (UniqueName: \"kubernetes.io/projected/589c5535-35b9-429d-bce9-34ad1ca25a7c-kube-api-access-kwxcg\") pod \"payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c\" (UID: \"589c5535-35b9-429d-bce9-34ad1ca25a7c\") " pod="openshift-ingress/payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c"
Oct 30 19:02:31.086970 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:31.086945    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589c5535-35b9-429d-bce9-34ad1ca25a7c-plugins-config-volume\") pod \"payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c\" (UID: \"589c5535-35b9-429d-bce9-34ad1ca25a7c\") " pod="openshift-ingress/payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c"
Oct 30 19:02:31.188477 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:31.188444    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589c5535-35b9-429d-bce9-34ad1ca25a7c-plugins-config-volume\") pod \"payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c\" (UID: \"589c5535-35b9-429d-bce9-34ad1ca25a7c\") " pod="openshift-ingress/payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c"
Oct 30 19:02:31.188672 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:31.188658    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwxcg\" (UniqueName: \"kubernetes.io/projected/589c5535-35b9-429d-bce9-34ad1ca25a7c-kube-api-access-kwxcg\") pod \"payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c\" (UID: \"589c5535-35b9-429d-bce9-34ad1ca25a7c\") " pod="openshift-ingress/payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c"
Oct 30 19:02:31.189674 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:31.189648    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589c5535-35b9-429d-bce9-34ad1ca25a7c-plugins-config-volume\") pod \"payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c\" (UID: \"589c5535-35b9-429d-bce9-34ad1ca25a7c\") " pod="openshift-ingress/payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c"
Oct 30 19:02:31.244941 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:31.244320    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwxcg\" (UniqueName: \"kubernetes.io/projected/589c5535-35b9-429d-bce9-34ad1ca25a7c-kube-api-access-kwxcg\") pod \"payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c\" (UID: \"589c5535-35b9-429d-bce9-34ad1ca25a7c\") " pod="openshift-ingress/payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c"
Oct 30 19:02:31.276593 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:31.276564    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:31.280086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:31.279588    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c"
Oct 30 19:02:31.528729 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:02:31.528697    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02fb175b_4f55_480a_a3d0_6ea00d85adaa.slice/crio-fbde321f05be8aef11b8897b1caa801f3fbd6ce09a7127226208543bebfb7f5b WatchSource:0}: Error finding container fbde321f05be8aef11b8897b1caa801f3fbd6ce09a7127226208543bebfb7f5b: Status 404 returned error can't find the container with id fbde321f05be8aef11b8897b1caa801f3fbd6ce09a7127226208543bebfb7f5b
Oct 30 19:02:31.530754 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:31.530732    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9"]
Oct 30 19:02:31.684741 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:02:31.684700    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod589c5535_35b9_429d_bce9_34ad1ca25a7c.slice/crio-0b0f64ff62103880fe17782b1e3e9ee5a69e28d62eba66d66079a34bf1ab3849 WatchSource:0}: Error finding container 0b0f64ff62103880fe17782b1e3e9ee5a69e28d62eba66d66079a34bf1ab3849: Status 404 returned error can't find the container with id 0b0f64ff62103880fe17782b1e3e9ee5a69e28d62eba66d66079a34bf1ab3849
Oct 30 19:02:31.686234 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:31.686212    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c"]
Oct 30 19:02:32.237016 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:32.236978    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9" event={"ID":"02fb175b-4f55-480a-a3d0-6ea00d85adaa","Type":"ContainerStarted","Data":"58009424e7076fdc7807bfacec8312822cd202104c89b37792869e97c5cf8884"}
Oct 30 19:02:32.237522 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:32.237033    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9" event={"ID":"02fb175b-4f55-480a-a3d0-6ea00d85adaa","Type":"ContainerStarted","Data":"fbde321f05be8aef11b8897b1caa801f3fbd6ce09a7127226208543bebfb7f5b"}
Oct 30 19:02:32.237522 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:32.237106    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9"
Oct 30 19:02:32.238849 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:32.238825    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c" event={"ID":"589c5535-35b9-429d-bce9-34ad1ca25a7c","Type":"ContainerStarted","Data":"548e8daea73606d849dbdab211414d59728accc3518671dd58553711b35c05e1"}
Oct 30 19:02:32.238985 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:32.238868    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c" event={"ID":"589c5535-35b9-429d-bce9-34ad1ca25a7c","Type":"ContainerStarted","Data":"0b0f64ff62103880fe17782b1e3e9ee5a69e28d62eba66d66079a34bf1ab3849"}
Oct 30 19:02:32.238985 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:32.238896    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c"
Oct 30 19:02:32.350273 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:32.350216    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9" podStartSLOduration=2.350202679 podStartE2EDuration="2.350202679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:32.344284345 +0000 UTC m=+1553.621334790" watchObservedRunningTime="2026-10-30 19:02:32.350202679 +0000 UTC m=+1553.627253124"
Oct 30 19:02:32.455813 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:32.455754    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c" podStartSLOduration=2.455739125 podStartE2EDuration="2.455739125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:32.451694189 +0000 UTC m=+1553.728744631" watchObservedRunningTime="2026-10-30 19:02:32.455739125 +0000 UTC m=+1553.732789593"
Oct 30 19:02:36.306946 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:36.306912    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:36.939974 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:36.939938    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:39.345680 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:39.345647    4924 scope.go:122] "RemoveContainer" containerID="77cef0164c1776a2ab88562814cbea8540494084c4cb68f28d9ddaa33f5ffd75"
Oct 30 19:02:39.365349 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:39.365322    4924 scope.go:122] "RemoveContainer" containerID="b149188310ded77d35a9afb536152b0fde44529ca9629e640ccfb823f3f5a215"
Oct 30 19:02:39.374865 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:39.374841    4924 scope.go:122] "RemoveContainer" containerID="990c33e4581f4d154128a37a69aa82f23e018114a49444cc2e519846f90db190"
Oct 30 19:02:39.384218 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:39.384198    4924 scope.go:122] "RemoveContainer" containerID="ac0cb6591f40e0094f978150d417ffc12e4acc5b9e9042a0e917e7ddfd2852f2"
Oct 30 19:02:41.318594 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:41.318557    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:42.050043 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:42.050008    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de3eb91-xtpb2"]
Oct 30 19:02:42.096463 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:42.096432    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de3eb91-xtpb2"]
Oct 30 19:02:42.848510 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:42.848474    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67125a81-e9f2-48c6-b024-0d98c501f112" path="/var/lib/kubelet/pods/67125a81-e9f2-48c6-b024-0d98c501f112/volumes"
Oct 30 19:02:43.266619 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:43.266534    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9"
Oct 30 19:02:43.268293 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:43.268269    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c"
Oct 30 19:02:45.892085 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:02:45.889226    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:02:46.138008 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:46.137910    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:49.741484 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:49.741402    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-948bc488-phkkb"]
Oct 30 19:02:49.746616 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:49.746594    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-948bc488-phkkb"
Oct 30 19:02:49.798935 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:49.798900    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bca2977-14c1-4b41-b784-6e93ab323aa5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-948bc488-phkkb\" (UID: \"1bca2977-14c1-4b41-b784-6e93ab323aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-948bc488-phkkb"
Oct 30 19:02:49.799106 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:49.798975    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pplm8\" (UniqueName: \"kubernetes.io/projected/1bca2977-14c1-4b41-b784-6e93ab323aa5-kube-api-access-pplm8\") pod \"maas-api-revoke-keys-e2e-migrate-948bc488-phkkb\" (UID: \"1bca2977-14c1-4b41-b784-6e93ab323aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-948bc488-phkkb"
Oct 30 19:02:49.799162 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:49.799114    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bca2977-14c1-4b41-b784-6e93ab323aa5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-948bc488-phkkb\" (UID: \"1bca2977-14c1-4b41-b784-6e93ab323aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-948bc488-phkkb"
Oct 30 19:02:49.799769 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:49.799745    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-948bc488-phkkb"]
Oct 30 19:02:49.840271 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:49.840237    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:02:49.840488 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:49.840467    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:02:49.842527 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:49.842506    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:02:49.842629 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:49.842529    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:02:49.900973 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:49.900944    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bca2977-14c1-4b41-b784-6e93ab323aa5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-948bc488-phkkb\" (UID: \"1bca2977-14c1-4b41-b784-6e93ab323aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-948bc488-phkkb"
Oct 30 19:02:49.901278 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:49.901005    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pplm8\" (UniqueName: \"kubernetes.io/projected/1bca2977-14c1-4b41-b784-6e93ab323aa5-kube-api-access-pplm8\") pod \"maas-api-revoke-keys-e2e-migrate-948bc488-phkkb\" (UID: \"1bca2977-14c1-4b41-b784-6e93ab323aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-948bc488-phkkb"
Oct 30 19:02:49.901278 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:49.901111    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bca2977-14c1-4b41-b784-6e93ab323aa5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-948bc488-phkkb\" (UID: \"1bca2977-14c1-4b41-b784-6e93ab323aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-948bc488-phkkb"
Oct 30 19:02:49.901823 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:49.901796    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bca2977-14c1-4b41-b784-6e93ab323aa5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-948bc488-phkkb\" (UID: \"1bca2977-14c1-4b41-b784-6e93ab323aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-948bc488-phkkb"
Oct 30 19:02:49.903643 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:49.903621    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bca2977-14c1-4b41-b784-6e93ab323aa5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-948bc488-phkkb\" (UID: \"1bca2977-14c1-4b41-b784-6e93ab323aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-948bc488-phkkb"
Oct 30 19:02:49.955648 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:49.955617    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pplm8\" (UniqueName: \"kubernetes.io/projected/1bca2977-14c1-4b41-b784-6e93ab323aa5-kube-api-access-pplm8\") pod \"maas-api-revoke-keys-e2e-migrate-948bc488-phkkb\" (UID: \"1bca2977-14c1-4b41-b784-6e93ab323aa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-948bc488-phkkb"
Oct 30 19:02:50.060695 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:50.060613    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-948bc488-phkkb"
Oct 30 19:02:50.433516 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:02:50.433488    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bca2977_14c1_4b41_b784_6e93ab323aa5.slice/crio-898adbd3bb86e767d02340f71c9765cc70fa36d38ea0e33fc455467b93a016af WatchSource:0}: Error finding container 898adbd3bb86e767d02340f71c9765cc70fa36d38ea0e33fc455467b93a016af: Status 404 returned error can't find the container with id 898adbd3bb86e767d02340f71c9765cc70fa36d38ea0e33fc455467b93a016af
Oct 30 19:02:50.434332 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:50.434310    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-948bc488-phkkb"]
Oct 30 19:02:50.463960 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:50.463894    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-948bc488-phkkb" event={"ID":"1bca2977-14c1-4b41-b784-6e93ab323aa5","Type":"ContainerStarted","Data":"898adbd3bb86e767d02340f71c9765cc70fa36d38ea0e33fc455467b93a016af"}
Oct 30 19:02:51.473783 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:51.473749    4924 generic.go:356] "Generic (PLEG): container finished" podID="1bca2977-14c1-4b41-b784-6e93ab323aa5" containerID="01686fbeec34af68c371f248740ee66d416de09aaa2b971e534dc0db12d1c8f9" exitCode=0
Oct 30 19:02:51.474223 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:51.473833    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-948bc488-phkkb" event={"ID":"1bca2977-14c1-4b41-b784-6e93ab323aa5","Type":"ContainerDied","Data":"01686fbeec34af68c371f248740ee66d416de09aaa2b971e534dc0db12d1c8f9"}
Oct 30 19:02:51.890872 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:51.890835    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:02:52.673360 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:52.673331    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-948bc488-phkkb"
Oct 30 19:02:52.742368 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:52.742278    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bca2977-14c1-4b41-b784-6e93ab323aa5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bca2977-14c1-4b41-b784-6e93ab323aa5-maas-api-xks-ca\") pod \"1bca2977-14c1-4b41-b784-6e93ab323aa5\" (UID: \"1bca2977-14c1-4b41-b784-6e93ab323aa5\") "
Oct 30 19:02:52.742533 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:52.742466    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bca2977-14c1-4b41-b784-6e93ab323aa5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bca2977-14c1-4b41-b784-6e93ab323aa5-maas-api-openshift-ca\") pod \"1bca2977-14c1-4b41-b784-6e93ab323aa5\" (UID: \"1bca2977-14c1-4b41-b784-6e93ab323aa5\") "
Oct 30 19:02:52.742660 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:52.742642    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bca2977-14c1-4b41-b784-6e93ab323aa5-kube-api-access-pplm8\" (UniqueName: \"kubernetes.io/projected/1bca2977-14c1-4b41-b784-6e93ab323aa5-kube-api-access-pplm8\") pod \"1bca2977-14c1-4b41-b784-6e93ab323aa5\" (UID: \"1bca2977-14c1-4b41-b784-6e93ab323aa5\") "
Oct 30 19:02:52.742829 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:52.742804    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bca2977-14c1-4b41-b784-6e93ab323aa5-maas-api-openshift-ca" pod "1bca2977-14c1-4b41-b784-6e93ab323aa5" (UID: "1bca2977-14c1-4b41-b784-6e93ab323aa5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:02:52.743213 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:52.743193    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bca2977-14c1-4b41-b784-6e93ab323aa5-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:52.744523 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:52.744499    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bca2977-14c1-4b41-b784-6e93ab323aa5-maas-api-xks-ca" pod "1bca2977-14c1-4b41-b784-6e93ab323aa5" (UID: "1bca2977-14c1-4b41-b784-6e93ab323aa5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:02:52.745022 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:52.745005    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bca2977-14c1-4b41-b784-6e93ab323aa5-kube-api-access-pplm8" pod "1bca2977-14c1-4b41-b784-6e93ab323aa5" (UID: "1bca2977-14c1-4b41-b784-6e93ab323aa5"). InnerVolumeSpecName "kube-api-access-pplm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:02:52.845237 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:52.845198    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pplm8\" (UniqueName: \"kubernetes.io/projected/1bca2977-14c1-4b41-b784-6e93ab323aa5-kube-api-access-pplm8\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:52.845237 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:52.845240    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bca2977-14c1-4b41-b784-6e93ab323aa5-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:53.491035 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:53.491004    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-948bc488-phkkb"
Oct 30 19:02:53.491241 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:53.491033    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-948bc488-phkkb" event={"ID":"1bca2977-14c1-4b41-b784-6e93ab323aa5","Type":"ContainerDied","Data":"898adbd3bb86e767d02340f71c9765cc70fa36d38ea0e33fc455467b93a016af"}
Oct 30 19:02:53.491241 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:53.491117    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="898adbd3bb86e767d02340f71c9765cc70fa36d38ea0e33fc455467b93a016af"
Oct 30 19:02:56.080122 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:56.079237    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b44d0e1-86vs5"]
Oct 30 19:02:56.089124 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:56.088674    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b44d0e1-86vs5"]
Oct 30 19:02:56.836680 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:56.836646    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca973523-e0ce-42d4-9d1e-bd0eab35c776" path="/var/lib/kubelet/pods/ca973523-e0ce-42d4-9d1e-bd0eab35c776/volumes"
Oct 30 19:02:57.017525 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:02:57.017460    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:00.769255 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:00.769222    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn"]
Oct 30 19:03:01.056370 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:01.056293    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:01.243756 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:01.243726    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:02.477614 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:02.477539    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:02.674172 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:02.674133    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:03.136195 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:03.136145    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n"]
Oct 30 19:03:03.138101 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:03.138056    4924 cpu_manager.go:414] "containerMap: removing container" podUID="1bca2977-14c1-4b41-b784-6e93ab323aa5" containerName="revoke-keys"
Oct 30 19:03:03.138251 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:03.138238    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bca2977-14c1-4b41-b784-6e93ab323aa5" containerName="revoke-keys"
Oct 30 19:03:03.138877 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:03.138856    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bca2977-14c1-4b41-b784-6e93ab323aa5" containerName="revoke-keys"
Oct 30 19:03:03.143010 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:03.142987    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n"
Oct 30 19:03:03.145926 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:03.145902    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42b85dc2-97ed-4a94-b7ad-4b2d45635e10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n\" (UID: \"42b85dc2-97ed-4a94-b7ad-4b2d45635e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n"
Oct 30 19:03:03.146017 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:03.145977    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42b85dc2-97ed-4a94-b7ad-4b2d45635e10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n\" (UID: \"42b85dc2-97ed-4a94-b7ad-4b2d45635e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n"
Oct 30 19:03:03.146284 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:03.146259    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w47b\" (UniqueName: \"kubernetes.io/projected/42b85dc2-97ed-4a94-b7ad-4b2d45635e10-kube-api-access-6w47b\") pod \"maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n\" (UID: \"42b85dc2-97ed-4a94-b7ad-4b2d45635e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n"
Oct 30 19:03:03.190697 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:03.190667    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n"]
Oct 30 19:03:03.230993 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:03.230960    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:03:03.231310 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:03.230972    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:03:03.231420 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:03.231025    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:03:03.240050 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:03.240024    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:03:03.247890 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:03.247853    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w47b\" (UniqueName: \"kubernetes.io/projected/42b85dc2-97ed-4a94-b7ad-4b2d45635e10-kube-api-access-6w47b\") pod \"maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n\" (UID: \"42b85dc2-97ed-4a94-b7ad-4b2d45635e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n"
Oct 30 19:03:03.248027 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:03.247966    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42b85dc2-97ed-4a94-b7ad-4b2d45635e10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n\" (UID: \"42b85dc2-97ed-4a94-b7ad-4b2d45635e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n"
Oct 30 19:03:03.248144 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:03.248104    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42b85dc2-97ed-4a94-b7ad-4b2d45635e10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n\" (UID: \"42b85dc2-97ed-4a94-b7ad-4b2d45635e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n"
Oct 30 19:03:03.248732 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:03.248704    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42b85dc2-97ed-4a94-b7ad-4b2d45635e10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n\" (UID: \"42b85dc2-97ed-4a94-b7ad-4b2d45635e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n"
Oct 30 19:03:03.250751 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:03.250727    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42b85dc2-97ed-4a94-b7ad-4b2d45635e10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n\" (UID: \"42b85dc2-97ed-4a94-b7ad-4b2d45635e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n"
Oct 30 19:03:03.302731 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:03.302687    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w47b\" (UniqueName: \"kubernetes.io/projected/42b85dc2-97ed-4a94-b7ad-4b2d45635e10-kube-api-access-6w47b\") pod \"maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n\" (UID: \"42b85dc2-97ed-4a94-b7ad-4b2d45635e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n"
Oct 30 19:03:03.456695 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:03.456604    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n"
Oct 30 19:03:03.827572 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:03:03.827542    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b85dc2_97ed_4a94_b7ad_4b2d45635e10.slice/crio-75cd3fe444b6469b4edc8e992a934b465d761dc071ae3ed0b38e235c25d31212 WatchSource:0}: Error finding container 75cd3fe444b6469b4edc8e992a934b465d761dc071ae3ed0b38e235c25d31212: Status 404 returned error can't find the container with id 75cd3fe444b6469b4edc8e992a934b465d761dc071ae3ed0b38e235c25d31212
Oct 30 19:03:03.829490 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:03.829465    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n"]
Oct 30 19:03:04.595764 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:04.595672    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n" event={"ID":"42b85dc2-97ed-4a94-b7ad-4b2d45635e10","Type":"ContainerStarted","Data":"b588edb5f3e457291b2e2577a73e2502023f492e2e21f98a9b020295a59c66f0"}
Oct 30 19:03:04.595764 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:04.595732    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n" event={"ID":"42b85dc2-97ed-4a94-b7ad-4b2d45635e10","Type":"ContainerStarted","Data":"75cd3fe444b6469b4edc8e992a934b465d761dc071ae3ed0b38e235c25d31212"}
Oct 30 19:03:04.709029 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:04.708977    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n" podStartSLOduration=1.708961881 podStartE2EDuration="1.708961881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:04.703385607 +0000 UTC m=+1585.980436052" watchObservedRunningTime="2026-10-30 19:03:04.708961881 +0000 UTC m=+1585.986012331"
Oct 30 19:03:05.605789 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:05.605757    4924 generic.go:356] "Generic (PLEG): container finished" podID="42b85dc2-97ed-4a94-b7ad-4b2d45635e10" containerID="b588edb5f3e457291b2e2577a73e2502023f492e2e21f98a9b020295a59c66f0" exitCode=0
Oct 30 19:03:05.606233 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:05.605843    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n" event={"ID":"42b85dc2-97ed-4a94-b7ad-4b2d45635e10","Type":"ContainerDied","Data":"b588edb5f3e457291b2e2577a73e2502023f492e2e21f98a9b020295a59c66f0"}
Oct 30 19:03:05.769865 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:05.769825    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn" podUID="4d01e582-b7f9-40a0-b166-13474d7f263c" containerName="payload-pre-processing" containerID="cri-o://64c93e5d3938ec04fe53faec8a4998d4aaa0966fbcfc3196a81f2b6189343d10" gracePeriod=25
Oct 30 19:03:06.077231 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.077201    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn"
Oct 30 19:03:06.200345 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.200261    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d01e582-b7f9-40a0-b166-13474d7f263c-kube-api-access-5hh5p\" (UniqueName: \"kubernetes.io/projected/4d01e582-b7f9-40a0-b166-13474d7f263c-kube-api-access-5hh5p\") pod \"4d01e582-b7f9-40a0-b166-13474d7f263c\" (UID: \"4d01e582-b7f9-40a0-b166-13474d7f263c\") "
Oct 30 19:03:06.200518 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.200493    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d01e582-b7f9-40a0-b166-13474d7f263c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d01e582-b7f9-40a0-b166-13474d7f263c-plugins-config-volume\") pod \"4d01e582-b7f9-40a0-b166-13474d7f263c\" (UID: \"4d01e582-b7f9-40a0-b166-13474d7f263c\") "
Oct 30 19:03:06.200963 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.200930    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d01e582-b7f9-40a0-b166-13474d7f263c-plugins-config-volume" pod "4d01e582-b7f9-40a0-b166-13474d7f263c" (UID: "4d01e582-b7f9-40a0-b166-13474d7f263c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:03:06.201356 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.201323    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d01e582-b7f9-40a0-b166-13474d7f263c-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:06.202802 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.202773    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d01e582-b7f9-40a0-b166-13474d7f263c-kube-api-access-5hh5p" pod "4d01e582-b7f9-40a0-b166-13474d7f263c" (UID: "4d01e582-b7f9-40a0-b166-13474d7f263c"). InnerVolumeSpecName "kube-api-access-5hh5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:03:06.302807 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.302762    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hh5p\" (UniqueName: \"kubernetes.io/projected/4d01e582-b7f9-40a0-b166-13474d7f263c-kube-api-access-5hh5p\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:06.615221 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.615180    4924 generic.go:356] "Generic (PLEG): container finished" podID="4d01e582-b7f9-40a0-b166-13474d7f263c" containerID="64c93e5d3938ec04fe53faec8a4998d4aaa0966fbcfc3196a81f2b6189343d10" exitCode=0
Oct 30 19:03:06.615708 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.615239    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn" event={"ID":"4d01e582-b7f9-40a0-b166-13474d7f263c","Type":"ContainerDied","Data":"64c93e5d3938ec04fe53faec8a4998d4aaa0966fbcfc3196a81f2b6189343d10"}
Oct 30 19:03:06.615708 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.615257    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn"
Oct 30 19:03:06.615708 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.615322    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn" event={"ID":"4d01e582-b7f9-40a0-b166-13474d7f263c","Type":"ContainerDied","Data":"b63762b9a5030ff529ecec24c5f3b664d7635a86900c6094afaa6f879679e2b9"}
Oct 30 19:03:06.615708 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.615349    4924 scope.go:122] "RemoveContainer" containerID="64c93e5d3938ec04fe53faec8a4998d4aaa0966fbcfc3196a81f2b6189343d10"
Oct 30 19:03:06.627989 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.627971    4924 scope.go:122] "RemoveContainer" containerID="64c93e5d3938ec04fe53faec8a4998d4aaa0966fbcfc3196a81f2b6189343d10"
Oct 30 19:03:06.628270 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:03:06.628252    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64c93e5d3938ec04fe53faec8a4998d4aaa0966fbcfc3196a81f2b6189343d10\": container with ID starting with 64c93e5d3938ec04fe53faec8a4998d4aaa0966fbcfc3196a81f2b6189343d10 not found: ID does not exist" containerID="64c93e5d3938ec04fe53faec8a4998d4aaa0966fbcfc3196a81f2b6189343d10"
Oct 30 19:03:06.628347 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.628278    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64c93e5d3938ec04fe53faec8a4998d4aaa0966fbcfc3196a81f2b6189343d10"} err="failed to get container status \"64c93e5d3938ec04fe53faec8a4998d4aaa0966fbcfc3196a81f2b6189343d10\": rpc error: code = NotFound desc = could not find container \"64c93e5d3938ec04fe53faec8a4998d4aaa0966fbcfc3196a81f2b6189343d10\": container with ID starting with 64c93e5d3938ec04fe53faec8a4998d4aaa0966fbcfc3196a81f2b6189343d10 not found: ID does not exist"
Oct 30 19:03:06.773680 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.773652    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n"
Oct 30 19:03:06.774809 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.774781    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn"]
Oct 30 19:03:06.809634 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.809607    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42b85dc2-97ed-4a94-b7ad-4b2d45635e10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42b85dc2-97ed-4a94-b7ad-4b2d45635e10-maas-api-openshift-ca\") pod \"42b85dc2-97ed-4a94-b7ad-4b2d45635e10\" (UID: \"42b85dc2-97ed-4a94-b7ad-4b2d45635e10\") "
Oct 30 19:03:06.809760 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.809721    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42b85dc2-97ed-4a94-b7ad-4b2d45635e10-kube-api-access-6w47b\" (UniqueName: \"kubernetes.io/projected/42b85dc2-97ed-4a94-b7ad-4b2d45635e10-kube-api-access-6w47b\") pod \"42b85dc2-97ed-4a94-b7ad-4b2d45635e10\" (UID: \"42b85dc2-97ed-4a94-b7ad-4b2d45635e10\") "
Oct 30 19:03:06.809916 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.809901    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42b85dc2-97ed-4a94-b7ad-4b2d45635e10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42b85dc2-97ed-4a94-b7ad-4b2d45635e10-maas-api-xks-ca\") pod \"42b85dc2-97ed-4a94-b7ad-4b2d45635e10\" (UID: \"42b85dc2-97ed-4a94-b7ad-4b2d45635e10\") "
Oct 30 19:03:06.810045 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.810012    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42b85dc2-97ed-4a94-b7ad-4b2d45635e10-maas-api-openshift-ca" pod "42b85dc2-97ed-4a94-b7ad-4b2d45635e10" (UID: "42b85dc2-97ed-4a94-b7ad-4b2d45635e10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:03:06.810584 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.810559    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42b85dc2-97ed-4a94-b7ad-4b2d45635e10-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:06.811830 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.811806    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b85dc2-97ed-4a94-b7ad-4b2d45635e10-kube-api-access-6w47b" pod "42b85dc2-97ed-4a94-b7ad-4b2d45635e10" (UID: "42b85dc2-97ed-4a94-b7ad-4b2d45635e10"). InnerVolumeSpecName "kube-api-access-6w47b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:03:06.811951 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.811927    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42b85dc2-97ed-4a94-b7ad-4b2d45635e10-maas-api-xks-ca" pod "42b85dc2-97ed-4a94-b7ad-4b2d45635e10" (UID: "42b85dc2-97ed-4a94-b7ad-4b2d45635e10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:03:06.820648 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.820617    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-948bc488-b694b88bc-926nn"]
Oct 30 19:03:06.838679 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.838651    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d01e582-b7f9-40a0-b166-13474d7f263c" path="/var/lib/kubelet/pods/4d01e582-b7f9-40a0-b166-13474d7f263c/volumes"
Oct 30 19:03:06.912289 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.912213    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w47b\" (UniqueName: \"kubernetes.io/projected/42b85dc2-97ed-4a94-b7ad-4b2d45635e10-kube-api-access-6w47b\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:06.912289 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:06.912255    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42b85dc2-97ed-4a94-b7ad-4b2d45635e10-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:07.559380 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:07.559348    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:07.635780 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:07.635748    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n"
Oct 30 19:03:07.635780 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:07.635767    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n" event={"ID":"42b85dc2-97ed-4a94-b7ad-4b2d45635e10","Type":"ContainerDied","Data":"75cd3fe444b6469b4edc8e992a934b465d761dc071ae3ed0b38e235c25d31212"}
Oct 30 19:03:07.636282 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:07.635828    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75cd3fe444b6469b4edc8e992a934b465d761dc071ae3ed0b38e235c25d31212"
Oct 30 19:03:09.459081 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:09.459032    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:09.945295 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:09.945263    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"]
Oct 30 19:03:09.946028 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:09.946006    4924 cpu_manager.go:414] "containerMap: removing container" podUID="4d01e582-b7f9-40a0-b166-13474d7f263c" containerName="payload-pre-processing"
Oct 30 19:03:09.946175 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:09.946028    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d01e582-b7f9-40a0-b166-13474d7f263c" containerName="payload-pre-processing"
Oct 30 19:03:09.946175 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:09.946082    4924 cpu_manager.go:414] "containerMap: removing container" podUID="42b85dc2-97ed-4a94-b7ad-4b2d45635e10" containerName="revoke-keys"
Oct 30 19:03:09.946175 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:09.946091    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42b85dc2-97ed-4a94-b7ad-4b2d45635e10" containerName="revoke-keys"
Oct 30 19:03:09.946361 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:09.946298    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d01e582-b7f9-40a0-b166-13474d7f263c" containerName="payload-pre-processing"
Oct 30 19:03:09.946361 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:09.946320    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="42b85dc2-97ed-4a94-b7ad-4b2d45635e10" containerName="revoke-keys"
Oct 30 19:03:09.949266 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:09.949244    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.015147 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.015114    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-3bd578-openshift-default-dockercfg-zrqpp\""
Oct 30 19:03:10.052501 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.052466    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"]
Oct 30 19:03:10.054858 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.054822    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-workload-socket\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.055169 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.055147    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-podinfo\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.055279 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.055180    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/571509d6-5172-43a0-9bc4-29a0da766ddc-istiod-ca-cert\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.055371 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.055355    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j86hb\" (UniqueName: \"kubernetes.io/projected/571509d6-5172-43a0-9bc4-29a0da766ddc-kube-api-access-j86hb\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.055434 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.055384    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-envoy\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.055540 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.055519    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-token\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.055600 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.055561    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-credential-socket\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.055600 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.055584    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-data\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.055703 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.055654    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-workload-certs\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.086433 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.086403    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:10.165093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.158039    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j86hb\" (UniqueName: \"kubernetes.io/projected/571509d6-5172-43a0-9bc4-29a0da766ddc-kube-api-access-j86hb\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.165093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.158122    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-envoy\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.165093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.158217    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-token\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.165093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.158247    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-credential-socket\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.165093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.158270    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-data\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.165093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.158347    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-workload-certs\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.165093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.158426    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-workload-socket\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.165093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.158516    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-podinfo\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.165093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.158545    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/571509d6-5172-43a0-9bc4-29a0da766ddc-istiod-ca-cert\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.165093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.159279    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-credential-socket\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.165093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.159548    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/571509d6-5172-43a0-9bc4-29a0da766ddc-istiod-ca-cert\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.165093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.159842    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-data\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.165093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.160017    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-workload-socket\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.165093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.160281    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-workload-certs\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.196935 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.196862    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-envoy\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.198189 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.198050    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-podinfo\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.215544 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.215509    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j86hb\" (UniqueName: \"kubernetes.io/projected/571509d6-5172-43a0-9bc4-29a0da766ddc-kube-api-access-j86hb\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.216540 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.216514    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-token\") pod \"e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") " pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.264078 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.264028    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:10.718364 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.718331    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:10.750934 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:03:10.748842    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod571509d6_5172_43a0_9bc4_29a0da766ddc.slice/crio-20527200146bc840c7a47625112942713facc9302979b1d53b236ea9c6defb6c WatchSource:0}: Error finding container 20527200146bc840c7a47625112942713facc9302979b1d53b236ea9c6defb6c: Status 404 returned error can't find the container with id 20527200146bc840c7a47625112942713facc9302979b1d53b236ea9c6defb6c
Oct 30 19:03:10.752396 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.752357    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"]
Oct 30 19:03:10.752514 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.752415    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:03:10.752514 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.752478    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:03:10.752640 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:10.752533    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:03:11.169752 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:11.169720    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8105ddbb-r2sfr"]
Oct 30 19:03:11.181049 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:11.181028    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8105ddbb-r2sfr"
Oct 30 19:03:11.229257 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:11.229227    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8105ddbb-r2sfr"]
Oct 30 19:03:11.230905 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:11.230881    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:03:11.231011 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:11.230990    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:03:11.231076 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:11.231023    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:03:11.231135 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:11.231109    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:03:11.285541 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:11.285512    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptng2\" (UniqueName: \"kubernetes.io/projected/9b5ba138-0529-4505-8ede-215d7e3c1b7a-kube-api-access-ptng2\") pod \"maas-api-revoke-sub-8105ddbb-r2sfr\" (UID: \"9b5ba138-0529-4505-8ede-215d7e3c1b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8105ddbb-r2sfr"
Oct 30 19:03:11.285707 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:11.285623    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b5ba138-0529-4505-8ede-215d7e3c1b7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8105ddbb-r2sfr\" (UID: \"9b5ba138-0529-4505-8ede-215d7e3c1b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8105ddbb-r2sfr"
Oct 30 19:03:11.285760 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:11.285716    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b5ba138-0529-4505-8ede-215d7e3c1b7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8105ddbb-r2sfr\" (UID: \"9b5ba138-0529-4505-8ede-215d7e3c1b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8105ddbb-r2sfr"
Oct 30 19:03:11.386815 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:11.386782    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptng2\" (UniqueName: \"kubernetes.io/projected/9b5ba138-0529-4505-8ede-215d7e3c1b7a-kube-api-access-ptng2\") pod \"maas-api-revoke-sub-8105ddbb-r2sfr\" (UID: \"9b5ba138-0529-4505-8ede-215d7e3c1b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8105ddbb-r2sfr"
Oct 30 19:03:11.387015 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:11.386858    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b5ba138-0529-4505-8ede-215d7e3c1b7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8105ddbb-r2sfr\" (UID: \"9b5ba138-0529-4505-8ede-215d7e3c1b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8105ddbb-r2sfr"
Oct 30 19:03:11.387015 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:11.386952    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b5ba138-0529-4505-8ede-215d7e3c1b7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8105ddbb-r2sfr\" (UID: \"9b5ba138-0529-4505-8ede-215d7e3c1b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8105ddbb-r2sfr"
Oct 30 19:03:11.387645 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:11.387619    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b5ba138-0529-4505-8ede-215d7e3c1b7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8105ddbb-r2sfr\" (UID: \"9b5ba138-0529-4505-8ede-215d7e3c1b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8105ddbb-r2sfr"
Oct 30 19:03:11.389208 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:11.389187    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b5ba138-0529-4505-8ede-215d7e3c1b7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8105ddbb-r2sfr\" (UID: \"9b5ba138-0529-4505-8ede-215d7e3c1b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8105ddbb-r2sfr"
Oct 30 19:03:11.441118 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:11.441019    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptng2\" (UniqueName: \"kubernetes.io/projected/9b5ba138-0529-4505-8ede-215d7e3c1b7a-kube-api-access-ptng2\") pod \"maas-api-revoke-sub-8105ddbb-r2sfr\" (UID: \"9b5ba138-0529-4505-8ede-215d7e3c1b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8105ddbb-r2sfr"
Oct 30 19:03:11.500857 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:11.500819    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8105ddbb-r2sfr"
Oct 30 19:03:11.674943 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:11.674903    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb" event={"ID":"571509d6-5172-43a0-9bc4-29a0da766ddc","Type":"ContainerStarted","Data":"0388ab91939968b207508aa89dca1f28c77e8bd0354457e0b3fb0a805aa76d89"}
Oct 30 19:03:11.675135 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:11.674967    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb" event={"ID":"571509d6-5172-43a0-9bc4-29a0da766ddc","Type":"ContainerStarted","Data":"20527200146bc840c7a47625112942713facc9302979b1d53b236ea9c6defb6c"}
Oct 30 19:03:11.787510 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:11.787417    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb" podStartSLOduration=2.787403638 podStartE2EDuration="2.787403638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:11.784587249 +0000 UTC m=+1593.061637693" watchObservedRunningTime="2026-10-30 19:03:11.787403638 +0000 UTC m=+1593.064454153"
Oct 30 19:03:11.959270 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:11.959240    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8105ddbb-r2sfr"]
Oct 30 19:03:12.248977 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.248946    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"]
Oct 30 19:03:12.253132 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.253113    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.264324 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.264301    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:12.265194 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.265169    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"]
Oct 30 19:03:12.269264 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.269246    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.270745 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.270728    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:12.344583 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.344553    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"]
Oct 30 19:03:12.386862 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.386827    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"]
Oct 30 19:03:12.404046 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.404014    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-credential-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.404239 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.404162    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztpvf\" (UniqueName: \"kubernetes.io/projected/dc3b282b-17c0-4f77-ba43-540f9524a8a9-kube-api-access-ztpvf\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.404239 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.404219    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c5b2946-7818-4067-976a-7e08e0f5c66c-istiod-ca-cert\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.404363 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.404333    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istiod-ca-cert\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.404427 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.404387    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-workload-certs\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.404502 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.404471    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-data\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.404556 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.404525    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-envoy\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.404609 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.404556    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-data\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.404609 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.404594    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-podinfo\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.404895 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.404641    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-workload-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.404895 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.404696    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-podinfo\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.404895 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.404737    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-credential-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.404895 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.404766    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w2v9\" (UniqueName: \"kubernetes.io/projected/4c5b2946-7818-4067-976a-7e08e0f5c66c-kube-api-access-6w2v9\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.404895 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.404787    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-token\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.404895 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.404852    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-workload-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.404895 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.404889    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-envoy\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.405299 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.404989    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-token\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.405299 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.405020    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-workload-certs\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.506314 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.506226    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istiod-ca-cert\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.506314 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.506303    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-workload-certs\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.506532 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.506329    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-data\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.506532 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.506348    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-envoy\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.506532 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.506372    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-data\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.506532 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.506409    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-podinfo\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.506532 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.506436    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-workload-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.506532 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.506476    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-podinfo\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.506532 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.506509    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-credential-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.506908 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.506552    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w2v9\" (UniqueName: \"kubernetes.io/projected/4c5b2946-7818-4067-976a-7e08e0f5c66c-kube-api-access-6w2v9\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.506908 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.506583    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-token\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.506908 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.506641    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-workload-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.506908 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.506668    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-envoy\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.506908 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.506707    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-token\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.506908 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.506735    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-workload-certs\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.506908 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.506785    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-data\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.506908 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.506853    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-credential-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.507350 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.506928    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztpvf\" (UniqueName: \"kubernetes.io/projected/dc3b282b-17c0-4f77-ba43-540f9524a8a9-kube-api-access-ztpvf\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.507350 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.506972    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c5b2946-7818-4067-976a-7e08e0f5c66c-istiod-ca-cert\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.507350 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.507101    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istiod-ca-cert\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.507350 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.507192    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-workload-certs\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.507573 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.507382    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-workload-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.507573 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.507434    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-credential-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.507764 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.507740    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-credential-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.507827 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.507760    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c5b2946-7818-4067-976a-7e08e0f5c66c-istiod-ca-cert\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.508375 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.508343    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-workload-socket\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.508616 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.508591    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-workload-certs\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.508884 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.508861    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-data\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.510236 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.510202    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-envoy\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.510350 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.510247    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-envoy\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.510935 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.510915    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-podinfo\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.511273 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.511251    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-podinfo\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.563776 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.563746    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-token\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.563988 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.563963    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztpvf\" (UniqueName: \"kubernetes.io/projected/dc3b282b-17c0-4f77-ba43-540f9524a8a9-kube-api-access-ztpvf\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.564142 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.564121    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-token\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.564199 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.564185    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w2v9\" (UniqueName: \"kubernetes.io/projected/4c5b2946-7818-4067-976a-7e08e0f5c66c-kube-api-access-6w2v9\") pod \"e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") " pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.566048 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.566015    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:12.582234 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.582212    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:12.688945 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.688661    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8105ddbb-r2sfr" event={"ID":"9b5ba138-0529-4505-8ede-215d7e3c1b7a","Type":"ContainerStarted","Data":"b74a61d7149264b02e947fb5f4683ffe19cff93a963687cf8c58abf1c8fd61e5"}
Oct 30 19:03:12.688945 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.688723    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8105ddbb-r2sfr" event={"ID":"9b5ba138-0529-4505-8ede-215d7e3c1b7a","Type":"ContainerStarted","Data":"b919eb8e6e822d6c19a0a396c7d8f811e30ab86c7ac638ff311ee2540f12cbff"}
Oct 30 19:03:12.688945 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.688751    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:12.689641 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.689605    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:03:12.801213 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.801114    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8105ddbb-r2sfr" podStartSLOduration=1.801097425 podStartE2EDuration="1.801097425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:12.796772264 +0000 UTC m=+1594.073822708" watchObservedRunningTime="2026-10-30 19:03:12.801097425 +0000 UTC m=+1594.078147877"
Oct 30 19:03:12.847385 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.844732    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:12.995802 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:03:12.995773    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3b282b_17c0_4f77_ba43_540f9524a8a9.slice/crio-fabfa151d09d2b894fbd27d1e3206509393e13d9771bb3c2d203172ce8509148 WatchSource:0}: Error finding container fabfa151d09d2b894fbd27d1e3206509393e13d9771bb3c2d203172ce8509148: Status 404 returned error can't find the container with id fabfa151d09d2b894fbd27d1e3206509393e13d9771bb3c2d203172ce8509148
Oct 30 19:03:12.997341 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.997315    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"]
Oct 30 19:03:12.998953 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.998919    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:03:12.999036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.998988    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:03:12.999036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:12.999024    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:03:13.003720 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:03:13.003695    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c5b2946_7818_4067_976a_7e08e0f5c66c.slice/crio-cfe967d0dfa5ca75e00fd75aa0cd79c5ebe940a630fc0b5a2685dbb3aa9fac6c WatchSource:0}: Error finding container cfe967d0dfa5ca75e00fd75aa0cd79c5ebe940a630fc0b5a2685dbb3aa9fac6c: Status 404 returned error can't find the container with id cfe967d0dfa5ca75e00fd75aa0cd79c5ebe940a630fc0b5a2685dbb3aa9fac6c
Oct 30 19:03:13.005454 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:13.005365    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"]
Oct 30 19:03:13.011719 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:13.011684    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:03:13.011799 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:13.011766    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:03:13.011856 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:13.011809    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:03:13.089732 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:03:13.089700    4924 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5ba138_0529_4505_8ede_215d7e3c1b7a.slice/crio-conmon-b74a61d7149264b02e947fb5f4683ffe19cff93a963687cf8c58abf1c8fd61e5.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:03:13.697949 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:13.697910    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4" event={"ID":"4c5b2946-7818-4067-976a-7e08e0f5c66c","Type":"ContainerStarted","Data":"06577ab5d460dc9cb1304d2c129618e503d42c52319d27061f982c77bf0b8e6a"}
Oct 30 19:03:13.698173 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:13.698009    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4" event={"ID":"4c5b2946-7818-4067-976a-7e08e0f5c66c","Type":"ContainerStarted","Data":"cfe967d0dfa5ca75e00fd75aa0cd79c5ebe940a630fc0b5a2685dbb3aa9fac6c"}
Oct 30 19:03:13.699615 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:13.699588    4924 generic.go:356] "Generic (PLEG): container finished" podID="9b5ba138-0529-4505-8ede-215d7e3c1b7a" containerID="b74a61d7149264b02e947fb5f4683ffe19cff93a963687cf8c58abf1c8fd61e5" exitCode=0
Oct 30 19:03:13.699746 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:13.699673    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8105ddbb-r2sfr" event={"ID":"9b5ba138-0529-4505-8ede-215d7e3c1b7a","Type":"ContainerDied","Data":"b74a61d7149264b02e947fb5f4683ffe19cff93a963687cf8c58abf1c8fd61e5"}
Oct 30 19:03:13.701434 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:13.701408    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964" event={"ID":"dc3b282b-17c0-4f77-ba43-540f9524a8a9","Type":"ContainerStarted","Data":"e0390dba402579c3422fdd1fcc4161e2162c4343de4e101656fcd2f3349f467a"}
Oct 30 19:03:13.701529 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:13.701450    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964" event={"ID":"dc3b282b-17c0-4f77-ba43-540f9524a8a9","Type":"ContainerStarted","Data":"fabfa151d09d2b894fbd27d1e3206509393e13d9771bb3c2d203172ce8509148"}
Oct 30 19:03:13.828088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:13.827271    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4" podStartSLOduration=1.8272516140000001 podStartE2EDuration="1.827251614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:13.825404757 +0000 UTC m=+1595.102455204" watchObservedRunningTime="2026-10-30 19:03:13.827251614 +0000 UTC m=+1595.104302078"
Oct 30 19:03:14.099503 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:14.098468    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964" podStartSLOduration=2.098449367 podStartE2EDuration="2.098449367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:14.098380673 +0000 UTC m=+1595.375431118" watchObservedRunningTime="2026-10-30 19:03:14.098449367 +0000 UTC m=+1595.375499815"
Oct 30 19:03:14.567177 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:14.567093    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:14.568925 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:14.568895    4924 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 19:03:14.569037 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:14.568947    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4" podUID="4c5b2946-7818-4067-976a-7e08e0f5c66c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 19:03:14.582700 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:14.582678    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:14.583975 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:14.583955    4924 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:03:14.584088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:14.583989    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964" podUID="dc3b282b-17c0-4f77-ba43-540f9524a8a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 19:03:14.584544 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:14.584526    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:14.673843 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:14.673810    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m"]
Oct 30 19:03:14.780126 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:14.780038    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:14.788008 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:14.787978    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb"]
Oct 30 19:03:14.969126 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:14.967654    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:15.084027 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:15.083989    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8105ddbb-r2sfr"
Oct 30 19:03:15.142137 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:15.101908    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b5ba138-0529-4505-8ede-215d7e3c1b7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b5ba138-0529-4505-8ede-215d7e3c1b7a-maas-api-openshift-ca\") pod \"9b5ba138-0529-4505-8ede-215d7e3c1b7a\" (UID: \"9b5ba138-0529-4505-8ede-215d7e3c1b7a\") "
Oct 30 19:03:15.142137 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:15.102146    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b5ba138-0529-4505-8ede-215d7e3c1b7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b5ba138-0529-4505-8ede-215d7e3c1b7a-maas-api-xks-ca\") pod \"9b5ba138-0529-4505-8ede-215d7e3c1b7a\" (UID: \"9b5ba138-0529-4505-8ede-215d7e3c1b7a\") "
Oct 30 19:03:15.142137 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:15.102291    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5ba138-0529-4505-8ede-215d7e3c1b7a-kube-api-access-ptng2\" (UniqueName: \"kubernetes.io/projected/9b5ba138-0529-4505-8ede-215d7e3c1b7a-kube-api-access-ptng2\") pod \"9b5ba138-0529-4505-8ede-215d7e3c1b7a\" (UID: \"9b5ba138-0529-4505-8ede-215d7e3c1b7a\") "
Oct 30 19:03:15.142137 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:15.126601    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5ba138-0529-4505-8ede-215d7e3c1b7a-kube-api-access-ptng2" pod "9b5ba138-0529-4505-8ede-215d7e3c1b7a" (UID: "9b5ba138-0529-4505-8ede-215d7e3c1b7a"). InnerVolumeSpecName "kube-api-access-ptng2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:03:15.142137 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:15.134573    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5ba138-0529-4505-8ede-215d7e3c1b7a-maas-api-openshift-ca" pod "9b5ba138-0529-4505-8ede-215d7e3c1b7a" (UID: "9b5ba138-0529-4505-8ede-215d7e3c1b7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:03:15.171278 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:15.170307    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b5ba138-0529-4505-8ede-215d7e3c1b7a-maas-api-xks-ca" pod "9b5ba138-0529-4505-8ede-215d7e3c1b7a" (UID: "9b5ba138-0529-4505-8ede-215d7e3c1b7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:03:15.176182 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:15.176145    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:15.219090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:15.211483    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b5ba138-0529-4505-8ede-215d7e3c1b7a-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:15.219090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:15.211519    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b5ba138-0529-4505-8ede-215d7e3c1b7a-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:15.219090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:15.211537    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptng2\" (UniqueName: \"kubernetes.io/projected/9b5ba138-0529-4505-8ede-215d7e3c1b7a-kube-api-access-ptng2\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:15.381354 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:15.381292    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:15.566484 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:15.566399    4924 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 19:03:15.566484 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:15.566448    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4" podUID="4c5b2946-7818-4067-976a-7e08e0f5c66c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 19:03:15.582942 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:15.582908    4924 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:03:15.583050 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:15.582947    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964" podUID="dc3b282b-17c0-4f77-ba43-540f9524a8a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 19:03:15.761263 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:03:15.761222    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:03:15.772712 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:15.772688    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8105ddbb-r2sfr"
Oct 30 19:03:15.772842 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:15.772731    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8105ddbb-r2sfr" event={"ID":"9b5ba138-0529-4505-8ede-215d7e3c1b7a","Type":"ContainerDied","Data":"b919eb8e6e822d6c19a0a396c7d8f811e30ab86c7ac638ff311ee2540f12cbff"}
Oct 30 19:03:15.772842 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:15.772824    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b919eb8e6e822d6c19a0a396c7d8f811e30ab86c7ac638ff311ee2540f12cbff"
Oct 30 19:03:16.572086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:16.571533    4924 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 19:03:16.572086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:16.571594    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4" podUID="4c5b2946-7818-4067-976a-7e08e0f5c66c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 19:03:16.585200 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:16.585164    4924 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:03:16.585498 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:16.585464    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964" podUID="dc3b282b-17c0-4f77-ba43-540f9524a8a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 19:03:17.567048 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:17.567006    4924 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 19:03:17.567241 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:17.567090    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4" podUID="4c5b2946-7818-4067-976a-7e08e0f5c66c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 19:03:17.582667 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:17.582640    4924 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:03:17.582996 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:17.582691    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964" podUID="dc3b282b-17c0-4f77-ba43-540f9524a8a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 19:03:18.463130 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:18.442227    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:18.577708 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:18.577020    4924 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 19:03:18.577708 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:18.577113    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4" podUID="4c5b2946-7818-4067-976a-7e08e0f5c66c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 19:03:18.583540 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:18.583510    4924 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:03:18.583883 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:18.583564    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964" podUID="dc3b282b-17c0-4f77-ba43-540f9524a8a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 19:03:18.676862 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:18.676810    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:18.870430 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:18.870392    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:18.936088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:18.933521    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf"]
Oct 30 19:03:18.936088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:18.934662    4924 cpu_manager.go:414] "containerMap: removing container" podUID="9b5ba138-0529-4505-8ede-215d7e3c1b7a" containerName="revoke-keys"
Oct 30 19:03:18.936088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:18.934683    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5ba138-0529-4505-8ede-215d7e3c1b7a" containerName="revoke-keys"
Oct 30 19:03:18.936088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:18.934944    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5ba138-0529-4505-8ede-215d7e3c1b7a" containerName="revoke-keys"
Oct 30 19:03:18.938960 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:18.938940    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf"
Oct 30 19:03:18.996682 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:18.992403    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-3bd578-dockercfg-c5n44\""
Oct 30 19:03:19.040214 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.017105    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-3bd578\""
Oct 30 19:03:19.042294 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.041647    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9nc9\" (UniqueName: \"kubernetes.io/projected/6e1e3aa8-965d-4de6-86ef-e10a83d01fed-kube-api-access-h9nc9\") pod \"payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf\" (UID: \"6e1e3aa8-965d-4de6-86ef-e10a83d01fed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf"
Oct 30 19:03:19.042294 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.041865    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e1e3aa8-965d-4de6-86ef-e10a83d01fed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf\" (UID: \"6e1e3aa8-965d-4de6-86ef-e10a83d01fed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf"
Oct 30 19:03:19.144280 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.143571    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9nc9\" (UniqueName: \"kubernetes.io/projected/6e1e3aa8-965d-4de6-86ef-e10a83d01fed-kube-api-access-h9nc9\") pod \"payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf\" (UID: \"6e1e3aa8-965d-4de6-86ef-e10a83d01fed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf"
Oct 30 19:03:19.144280 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.143770    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e1e3aa8-965d-4de6-86ef-e10a83d01fed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf\" (UID: \"6e1e3aa8-965d-4de6-86ef-e10a83d01fed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf"
Oct 30 19:03:19.149693 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.149662    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e1e3aa8-965d-4de6-86ef-e10a83d01fed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf\" (UID: \"6e1e3aa8-965d-4de6-86ef-e10a83d01fed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf"
Oct 30 19:03:19.168113 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.163304    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf"]
Oct 30 19:03:19.228289 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.217339    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr"]
Oct 30 19:03:19.250487 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.249361    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr"
Oct 30 19:03:19.265475 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.265442    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr"]
Oct 30 19:03:19.317727 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.316264    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9nc9\" (UniqueName: \"kubernetes.io/projected/6e1e3aa8-965d-4de6-86ef-e10a83d01fed-kube-api-access-h9nc9\") pod \"payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf\" (UID: \"6e1e3aa8-965d-4de6-86ef-e10a83d01fed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf"
Oct 30 19:03:19.355107 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.355048    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16e8a3e-7776-4313-a122-7ff9644f74d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr\" (UID: \"b16e8a3e-7776-4313-a122-7ff9644f74d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr"
Oct 30 19:03:19.355558 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.355526    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrswz\" (UniqueName: \"kubernetes.io/projected/b16e8a3e-7776-4313-a122-7ff9644f74d7-kube-api-access-mrswz\") pod \"payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr\" (UID: \"b16e8a3e-7776-4313-a122-7ff9644f74d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr"
Oct 30 19:03:19.416473 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.416313    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:19.459013 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.457541    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16e8a3e-7776-4313-a122-7ff9644f74d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr\" (UID: \"b16e8a3e-7776-4313-a122-7ff9644f74d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr"
Oct 30 19:03:19.459013 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.457901    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrswz\" (UniqueName: \"kubernetes.io/projected/b16e8a3e-7776-4313-a122-7ff9644f74d7-kube-api-access-mrswz\") pod \"payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr\" (UID: \"b16e8a3e-7776-4313-a122-7ff9644f74d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr"
Oct 30 19:03:19.459013 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.458675    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16e8a3e-7776-4313-a122-7ff9644f74d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr\" (UID: \"b16e8a3e-7776-4313-a122-7ff9644f74d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr"
Oct 30 19:03:19.513297 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.513260    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrswz\" (UniqueName: \"kubernetes.io/projected/b16e8a3e-7776-4313-a122-7ff9644f74d7-kube-api-access-mrswz\") pod \"payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr\" (UID: \"b16e8a3e-7776-4313-a122-7ff9644f74d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr"
Oct 30 19:03:19.562028 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.561998    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf"
Oct 30 19:03:19.566429 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.566393    4924 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 19:03:19.566550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.566455    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4" podUID="4c5b2946-7818-4067-976a-7e08e0f5c66c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 19:03:19.578169 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.578100    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr"
Oct 30 19:03:19.584519 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.584486    4924 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 19:03:19.585196 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.585016    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964" podUID="dc3b282b-17c0-4f77-ba43-540f9524a8a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 19:03:19.676486 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.675878    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m" podUID="a1f3722a-e299-4178-a91f-1fa91e6014f6" containerName="payload-processing" containerID="cri-o://b1902f135678ef6c20b33e7feff98eea17dce4bb5089937b44f1bc365127fc59" gracePeriod=25
Oct 30 19:03:19.789178 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.789130    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb" podUID="d1a29457-36fd-4c0f-b19f-80fab8e0725a" containerName="payload-pre-processing" containerID="cri-o://e09d590b1ed33ecc83314a8363b6b4815af6aa296790d870cfe78c2a4b1c60f3" gracePeriod=25
Oct 30 19:03:19.996393 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:19.996367    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m"
Oct 30 19:03:20.030934 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.030902    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf"]
Oct 30 19:03:20.069879 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.068553    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1f3722a-e299-4178-a91f-1fa91e6014f6-kube-api-access-hjv5v\" (UniqueName: \"kubernetes.io/projected/a1f3722a-e299-4178-a91f-1fa91e6014f6-kube-api-access-hjv5v\") pod \"a1f3722a-e299-4178-a91f-1fa91e6014f6\" (UID: \"a1f3722a-e299-4178-a91f-1fa91e6014f6\") "
Oct 30 19:03:20.069879 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.068720    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1f3722a-e299-4178-a91f-1fa91e6014f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1f3722a-e299-4178-a91f-1fa91e6014f6-plugins-config-volume\") pod \"a1f3722a-e299-4178-a91f-1fa91e6014f6\" (UID: \"a1f3722a-e299-4178-a91f-1fa91e6014f6\") "
Oct 30 19:03:20.071021 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.070634    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1f3722a-e299-4178-a91f-1fa91e6014f6-plugins-config-volume" pod "a1f3722a-e299-4178-a91f-1fa91e6014f6" (UID: "a1f3722a-e299-4178-a91f-1fa91e6014f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:03:20.084548 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.084303    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1f3722a-e299-4178-a91f-1fa91e6014f6-kube-api-access-hjv5v" pod "a1f3722a-e299-4178-a91f-1fa91e6014f6" (UID: "a1f3722a-e299-4178-a91f-1fa91e6014f6"). InnerVolumeSpecName "kube-api-access-hjv5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:03:20.102541 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.102393    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr"]
Oct 30 19:03:20.143276 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.143155    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe40c14f-kfprk"]
Oct 30 19:03:20.162852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.162112    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe40c14f-kfprk"]
Oct 30 19:03:20.170165 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.170138    4924 generic.go:356] "Generic (PLEG): container finished" podID="a1f3722a-e299-4178-a91f-1fa91e6014f6" containerID="b1902f135678ef6c20b33e7feff98eea17dce4bb5089937b44f1bc365127fc59" exitCode=0
Oct 30 19:03:20.170430 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.170412    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m" event={"ID":"a1f3722a-e299-4178-a91f-1fa91e6014f6","Type":"ContainerDied","Data":"b1902f135678ef6c20b33e7feff98eea17dce4bb5089937b44f1bc365127fc59"}
Oct 30 19:03:20.170553 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.170541    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m" event={"ID":"a1f3722a-e299-4178-a91f-1fa91e6014f6","Type":"ContainerDied","Data":"ed985f74a69eae40e586e3459a9203349d94c5db51f197813c8388338beee6bb"}
Oct 30 19:03:20.170649 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.170639    4924 scope.go:122] "RemoveContainer" containerID="b1902f135678ef6c20b33e7feff98eea17dce4bb5089937b44f1bc365127fc59"
Oct 30 19:03:20.170934 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.170892    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m"
Oct 30 19:03:20.174519 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.174489    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1f3722a-e299-4178-a91f-1fa91e6014f6-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:20.174612 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.174525    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjv5v\" (UniqueName: \"kubernetes.io/projected/a1f3722a-e299-4178-a91f-1fa91e6014f6-kube-api-access-hjv5v\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:20.180139 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.179681    4924 generic.go:356] "Generic (PLEG): container finished" podID="d1a29457-36fd-4c0f-b19f-80fab8e0725a" containerID="e09d590b1ed33ecc83314a8363b6b4815af6aa296790d870cfe78c2a4b1c60f3" exitCode=0
Oct 30 19:03:20.180139 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.179851    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb" event={"ID":"d1a29457-36fd-4c0f-b19f-80fab8e0725a","Type":"ContainerDied","Data":"e09d590b1ed33ecc83314a8363b6b4815af6aa296790d870cfe78c2a4b1c60f3"}
Oct 30 19:03:20.185097 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.185014    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr" event={"ID":"b16e8a3e-7776-4313-a122-7ff9644f74d7","Type":"ContainerStarted","Data":"cf627786042e3ebe77ede1c3bf3b69e3c2aa0c03b4740ed783258a16c64872e7"}
Oct 30 19:03:20.187284 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.187257    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf" event={"ID":"6e1e3aa8-965d-4de6-86ef-e10a83d01fed","Type":"ContainerStarted","Data":"4b579b20576c38132f02bc6d5631c5cb7021c6dbfe489be299769f638815b00f"}
Oct 30 19:03:20.210239 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.210215    4924 scope.go:122] "RemoveContainer" containerID="b1902f135678ef6c20b33e7feff98eea17dce4bb5089937b44f1bc365127fc59"
Oct 30 19:03:20.210586 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:03:20.210553    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1902f135678ef6c20b33e7feff98eea17dce4bb5089937b44f1bc365127fc59\": container with ID starting with b1902f135678ef6c20b33e7feff98eea17dce4bb5089937b44f1bc365127fc59 not found: ID does not exist" containerID="b1902f135678ef6c20b33e7feff98eea17dce4bb5089937b44f1bc365127fc59"
Oct 30 19:03:20.210687 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.210591    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1902f135678ef6c20b33e7feff98eea17dce4bb5089937b44f1bc365127fc59"} err="failed to get container status \"b1902f135678ef6c20b33e7feff98eea17dce4bb5089937b44f1bc365127fc59\": rpc error: code = NotFound desc = could not find container \"b1902f135678ef6c20b33e7feff98eea17dce4bb5089937b44f1bc365127fc59\": container with ID starting with b1902f135678ef6c20b33e7feff98eea17dce4bb5089937b44f1bc365127fc59 not found: ID does not exist"
Oct 30 19:03:20.224395 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.224305    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb"
Oct 30 19:03:20.275962 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.275886    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1a29457-36fd-4c0f-b19f-80fab8e0725a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1a29457-36fd-4c0f-b19f-80fab8e0725a-plugins-config-volume\") pod \"d1a29457-36fd-4c0f-b19f-80fab8e0725a\" (UID: \"d1a29457-36fd-4c0f-b19f-80fab8e0725a\") "
Oct 30 19:03:20.276146 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.276022    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1a29457-36fd-4c0f-b19f-80fab8e0725a-kube-api-access-tsvdr\" (UniqueName: \"kubernetes.io/projected/d1a29457-36fd-4c0f-b19f-80fab8e0725a-kube-api-access-tsvdr\") pod \"d1a29457-36fd-4c0f-b19f-80fab8e0725a\" (UID: \"d1a29457-36fd-4c0f-b19f-80fab8e0725a\") "
Oct 30 19:03:20.276335 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.276309    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1a29457-36fd-4c0f-b19f-80fab8e0725a-plugins-config-volume" pod "d1a29457-36fd-4c0f-b19f-80fab8e0725a" (UID: "d1a29457-36fd-4c0f-b19f-80fab8e0725a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:03:20.276791 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.276768    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1a29457-36fd-4c0f-b19f-80fab8e0725a-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:20.278009 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.277988    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a29457-36fd-4c0f-b19f-80fab8e0725a-kube-api-access-tsvdr" pod "d1a29457-36fd-4c0f-b19f-80fab8e0725a" (UID: "d1a29457-36fd-4c0f-b19f-80fab8e0725a"). InnerVolumeSpecName "kube-api-access-tsvdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:03:20.340705 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.340684    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m"]
Oct 30 19:03:20.349628 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.349602    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-85b6ee7b-54f6b4cff4-vhr8m"]
Oct 30 19:03:20.378107 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.378078    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsvdr\" (UniqueName: \"kubernetes.io/projected/d1a29457-36fd-4c0f-b19f-80fab8e0725a-kube-api-access-tsvdr\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:20.845084 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.844438    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ae49714-1974-4577-b226-b6a0189673ed" path="/var/lib/kubelet/pods/5ae49714-1974-4577-b226-b6a0189673ed/volumes"
Oct 30 19:03:20.845691 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:20.845111    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1f3722a-e299-4178-a91f-1fa91e6014f6" path="/var/lib/kubelet/pods/a1f3722a-e299-4178-a91f-1fa91e6014f6/volumes"
Oct 30 19:03:20.886084 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:03:20.884639    4924 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a29457_36fd_4c0f_b19f_80fab8e0725a.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 19:03:21.124569 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:21.124382    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:21.125352 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:21.125098    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:21.126195 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:21.126173    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:03:21.165971 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:21.165942    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:21.167352 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:21.167327    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:21.167921 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:21.167900    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:03:21.198951 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:21.198914    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf" event={"ID":"6e1e3aa8-965d-4de6-86ef-e10a83d01fed","Type":"ContainerStarted","Data":"279f6e55cad0dceec45e7b300917d4eda403a72571b47e8b6aa98e568f6e53d5"}
Oct 30 19:03:21.199169 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:21.199000    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf"
Oct 30 19:03:21.201549 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:21.201529    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb"
Oct 30 19:03:21.201693 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:21.201572    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb" event={"ID":"d1a29457-36fd-4c0f-b19f-80fab8e0725a","Type":"ContainerDied","Data":"29993da22c5d48d2f2e5566f4abeda3771a9b738ff4af00155d0e451ec7c28da"}
Oct 30 19:03:21.201693 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:21.201617    4924 scope.go:122] "RemoveContainer" containerID="e09d590b1ed33ecc83314a8363b6b4815af6aa296790d870cfe78c2a4b1c60f3"
Oct 30 19:03:21.203814 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:21.203748    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr" event={"ID":"b16e8a3e-7776-4313-a122-7ff9644f74d7","Type":"ContainerStarted","Data":"1239c6b142b9875e046f26e96afcb591e2a3c233abaf535b61d92f6fcaf7e37a"}
Oct 30 19:03:21.204158 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:21.204137    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr"
Oct 30 19:03:21.345051 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:21.345017    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb"]
Oct 30 19:03:21.350141 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:21.350115    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-85b6ee7b-6c75b967c9-nxsdb"]
Oct 30 19:03:21.451266 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:21.451167    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr" podStartSLOduration=2.4511508 podStartE2EDuration="2.4511508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:21.44983461 +0000 UTC m=+1602.726885055" watchObservedRunningTime="2026-10-30 19:03:21.4511508 +0000 UTC m=+1602.728201246"
Oct 30 19:03:21.669275 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:21.669209    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf" podStartSLOduration=3.669189679 podStartE2EDuration="3.669189679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:21.66739334 +0000 UTC m=+1602.944443784" watchObservedRunningTime="2026-10-30 19:03:21.669189679 +0000 UTC m=+1602.946240125"
Oct 30 19:03:22.837008 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:22.836969    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1a29457-36fd-4c0f-b19f-80fab8e0725a" path="/var/lib/kubelet/pods/d1a29457-36fd-4c0f-b19f-80fab8e0725a/volumes"
Oct 30 19:03:23.731217 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:23.731184    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:23.945377 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:23.945343    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:24.146910 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:24.146881    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:26.873082 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:26.870565    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:27.064485 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:27.064447    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:27.333484 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:27.333374    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9"]
Oct 30 19:03:27.334699 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:27.334676    4924 cpu_manager.go:414] "containerMap: removing container" podUID="d1a29457-36fd-4c0f-b19f-80fab8e0725a" containerName="payload-pre-processing"
Oct 30 19:03:27.334820 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:27.334810    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1a29457-36fd-4c0f-b19f-80fab8e0725a" containerName="payload-pre-processing"
Oct 30 19:03:27.334879 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:27.334873    4924 cpu_manager.go:414] "containerMap: removing container" podUID="a1f3722a-e299-4178-a91f-1fa91e6014f6" containerName="payload-processing"
Oct 30 19:03:27.334932 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:27.334925    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1f3722a-e299-4178-a91f-1fa91e6014f6" containerName="payload-processing"
Oct 30 19:03:27.335264 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:27.335192    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1a29457-36fd-4c0f-b19f-80fab8e0725a" containerName="payload-pre-processing"
Oct 30 19:03:27.335264 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:27.335229    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1f3722a-e299-4178-a91f-1fa91e6014f6" containerName="payload-processing"
Oct 30 19:03:27.339314 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:27.339297    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9"
Oct 30 19:03:27.411323 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:27.411283    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9"]
Oct 30 19:03:27.433456 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:27.433421    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:03:27.433873 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:27.433843    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:03:27.434253 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:27.434237    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:03:27.448036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:27.448018    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:03:27.458680 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:27.458627    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6szf2\" (UniqueName: \"kubernetes.io/projected/b7be7cc1-a8ba-4176-8aeb-e337d524e43a-kube-api-access-6szf2\") pod \"maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9\" (UID: \"b7be7cc1-a8ba-4176-8aeb-e337d524e43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9"
Oct 30 19:03:27.459097 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:27.459059    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7be7cc1-a8ba-4176-8aeb-e337d524e43a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9\" (UID: \"b7be7cc1-a8ba-4176-8aeb-e337d524e43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9"
Oct 30 19:03:27.459361 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:27.459340    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7be7cc1-a8ba-4176-8aeb-e337d524e43a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9\" (UID: \"b7be7cc1-a8ba-4176-8aeb-e337d524e43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9"
Oct 30 19:03:27.549135 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:27.539628    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:27.646948 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:27.646911    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7be7cc1-a8ba-4176-8aeb-e337d524e43a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9\" (UID: \"b7be7cc1-a8ba-4176-8aeb-e337d524e43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9"
Oct 30 19:03:27.647594 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:27.647569    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7be7cc1-a8ba-4176-8aeb-e337d524e43a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9\" (UID: \"b7be7cc1-a8ba-4176-8aeb-e337d524e43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9"
Oct 30 19:03:27.647887 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:27.647866    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7be7cc1-a8ba-4176-8aeb-e337d524e43a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9\" (UID: \"b7be7cc1-a8ba-4176-8aeb-e337d524e43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9"
Oct 30 19:03:27.648800 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:27.648776    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6szf2\" (UniqueName: \"kubernetes.io/projected/b7be7cc1-a8ba-4176-8aeb-e337d524e43a-kube-api-access-6szf2\") pod \"maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9\" (UID: \"b7be7cc1-a8ba-4176-8aeb-e337d524e43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9"
Oct 30 19:03:27.699390 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:27.699364    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7be7cc1-a8ba-4176-8aeb-e337d524e43a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9\" (UID: \"b7be7cc1-a8ba-4176-8aeb-e337d524e43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9"
Oct 30 19:03:27.758089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:27.730045    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6szf2\" (UniqueName: \"kubernetes.io/projected/b7be7cc1-a8ba-4176-8aeb-e337d524e43a-kube-api-access-6szf2\") pod \"maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9\" (UID: \"b7be7cc1-a8ba-4176-8aeb-e337d524e43a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9"
Oct 30 19:03:27.960440 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:27.960350    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9"
Oct 30 19:03:28.756360 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:03:28.755593    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7be7cc1_a8ba_4176_8aeb_e337d524e43a.slice/crio-a6eb36ef4cd382702e90ae42127c33c9a5870041045db7e9bc9c9adf18a4451e WatchSource:0}: Error finding container a6eb36ef4cd382702e90ae42127c33c9a5870041045db7e9bc9c9adf18a4451e: Status 404 returned error can't find the container with id a6eb36ef4cd382702e90ae42127c33c9a5870041045db7e9bc9c9adf18a4451e
Oct 30 19:03:28.756603 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:28.756574    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9"]
Oct 30 19:03:29.339187 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:29.339039    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:29.388723 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:29.388638    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9" event={"ID":"b7be7cc1-a8ba-4176-8aeb-e337d524e43a","Type":"ContainerStarted","Data":"7d53251a516c6b0f98b2602dc56c4bceb82481045ca48f283638f897f72df55f"}
Oct 30 19:03:29.388915 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:29.388795    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9" event={"ID":"b7be7cc1-a8ba-4176-8aeb-e337d524e43a","Type":"ContainerStarted","Data":"a6eb36ef4cd382702e90ae42127c33c9a5870041045db7e9bc9c9adf18a4451e"}
Oct 30 19:03:29.500347 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:29.500296    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9" podStartSLOduration=2.500282517 podStartE2EDuration="2.500282517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:29.495938813 +0000 UTC m=+1610.772989258" watchObservedRunningTime="2026-10-30 19:03:29.500282517 +0000 UTC m=+1610.777332962"
Oct 30 19:03:30.398958 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:30.398922    4924 generic.go:356] "Generic (PLEG): container finished" podID="b7be7cc1-a8ba-4176-8aeb-e337d524e43a" containerID="7d53251a516c6b0f98b2602dc56c4bceb82481045ca48f283638f897f72df55f" exitCode=0
Oct 30 19:03:30.399373 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:30.399049    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9" event={"ID":"b7be7cc1-a8ba-4176-8aeb-e337d524e43a","Type":"ContainerDied","Data":"7d53251a516c6b0f98b2602dc56c4bceb82481045ca48f283638f897f72df55f"}
Oct 30 19:03:31.598403 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:31.598381    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9"
Oct 30 19:03:31.657577 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:31.657542    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7be7cc1-a8ba-4176-8aeb-e337d524e43a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7be7cc1-a8ba-4176-8aeb-e337d524e43a-maas-api-openshift-ca\") pod \"b7be7cc1-a8ba-4176-8aeb-e337d524e43a\" (UID: \"b7be7cc1-a8ba-4176-8aeb-e337d524e43a\") "
Oct 30 19:03:31.657780 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:31.657624    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7be7cc1-a8ba-4176-8aeb-e337d524e43a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7be7cc1-a8ba-4176-8aeb-e337d524e43a-maas-api-xks-ca\") pod \"b7be7cc1-a8ba-4176-8aeb-e337d524e43a\" (UID: \"b7be7cc1-a8ba-4176-8aeb-e337d524e43a\") "
Oct 30 19:03:31.657877 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:31.657839    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7be7cc1-a8ba-4176-8aeb-e337d524e43a-kube-api-access-6szf2\" (UniqueName: \"kubernetes.io/projected/b7be7cc1-a8ba-4176-8aeb-e337d524e43a-kube-api-access-6szf2\") pod \"b7be7cc1-a8ba-4176-8aeb-e337d524e43a\" (UID: \"b7be7cc1-a8ba-4176-8aeb-e337d524e43a\") "
Oct 30 19:03:31.657940 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:31.657912    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7be7cc1-a8ba-4176-8aeb-e337d524e43a-maas-api-openshift-ca" pod "b7be7cc1-a8ba-4176-8aeb-e337d524e43a" (UID: "b7be7cc1-a8ba-4176-8aeb-e337d524e43a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:03:31.658506 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:31.658451    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7be7cc1-a8ba-4176-8aeb-e337d524e43a-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:31.659765 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:31.659737    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7be7cc1-a8ba-4176-8aeb-e337d524e43a-maas-api-xks-ca" pod "b7be7cc1-a8ba-4176-8aeb-e337d524e43a" (UID: "b7be7cc1-a8ba-4176-8aeb-e337d524e43a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:03:31.659917 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:31.659886    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7be7cc1-a8ba-4176-8aeb-e337d524e43a-kube-api-access-6szf2" pod "b7be7cc1-a8ba-4176-8aeb-e337d524e43a" (UID: "b7be7cc1-a8ba-4176-8aeb-e337d524e43a"). InnerVolumeSpecName "kube-api-access-6szf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:03:31.760869 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:31.760833    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7be7cc1-a8ba-4176-8aeb-e337d524e43a-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:31.760869 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:31.760872    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6szf2\" (UniqueName: \"kubernetes.io/projected/b7be7cc1-a8ba-4176-8aeb-e337d524e43a-kube-api-access-6szf2\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:32.216511 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:32.216482    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr"
Oct 30 19:03:32.216673 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:32.216554    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf"
Oct 30 19:03:32.418577 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:32.418538    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9" event={"ID":"b7be7cc1-a8ba-4176-8aeb-e337d524e43a","Type":"ContainerDied","Data":"a6eb36ef4cd382702e90ae42127c33c9a5870041045db7e9bc9c9adf18a4451e"}
Oct 30 19:03:32.418762 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:32.418604    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6eb36ef4cd382702e90ae42127c33c9a5870041045db7e9bc9c9adf18a4451e"
Oct 30 19:03:32.418762 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:32.418553    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9"
Oct 30 19:03:33.618477 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:33.618444    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:33.863289 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:33.863255    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:34.049415 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.049341    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034fccb1-rgxw7"]
Oct 30 19:03:34.095973 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.095942    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034fccb1-rgxw7"]
Oct 30 19:03:34.148584 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.148551    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:34.595862 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.595828    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"]
Oct 30 19:03:34.597357 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.596813    4924 cpu_manager.go:414] "containerMap: removing container" podUID="b7be7cc1-a8ba-4176-8aeb-e337d524e43a" containerName="revoke-keys"
Oct 30 19:03:34.597357 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.596838    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7be7cc1-a8ba-4176-8aeb-e337d524e43a" containerName="revoke-keys"
Oct 30 19:03:34.597357 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.597055    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7be7cc1-a8ba-4176-8aeb-e337d524e43a" containerName="revoke-keys"
Oct 30 19:03:34.600744 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.600725    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.668311 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.662382    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-8be536-openshift-default-dockercfg-cfmdr\""
Oct 30 19:03:34.668780 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.668721    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:34.726212 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.726155    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"]
Oct 30 19:03:34.739229 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.739187    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-istio-envoy\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.739419 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.739361    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-workload-socket\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.739501 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.739428    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-istio-data\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.739563 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.739504    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ef253e-1870-4b85-ab78-f155131276a9-istiod-ca-cert\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.739563 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.739556    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ef253e-1870-4b85-ab78-f155131276a9-istio-token\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.739670 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.739659    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ef253e-1870-4b85-ab78-f155131276a9-istio-podinfo\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.739824 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.739804    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lts6d\" (UniqueName: \"kubernetes.io/projected/91ef253e-1870-4b85-ab78-f155131276a9-kube-api-access-lts6d\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.740082 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.740048    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-credential-socket\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.740183 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.740111    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-workload-certs\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.839468 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.839433    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="052fcea7-50f4-444b-ad19-6c7346019e00" path="/var/lib/kubelet/pods/052fcea7-50f4-444b-ad19-6c7346019e00/volumes"
Oct 30 19:03:34.841082 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.841044    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-istio-envoy\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.841209 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.841189    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-workload-socket\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.841274 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.841240    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-istio-data\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.841332 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.841296    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ef253e-1870-4b85-ab78-f155131276a9-istiod-ca-cert\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.841384 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.841328    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ef253e-1870-4b85-ab78-f155131276a9-istio-token\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.841449 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.841412    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ef253e-1870-4b85-ab78-f155131276a9-istio-podinfo\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.841547 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.841526    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lts6d\" (UniqueName: \"kubernetes.io/projected/91ef253e-1870-4b85-ab78-f155131276a9-kube-api-access-lts6d\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.841759 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.841718    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-workload-socket\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.842559 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.842522    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-istio-data\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.842686 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.842647    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ef253e-1870-4b85-ab78-f155131276a9-istiod-ca-cert\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.842820 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.842793    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-credential-socket\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.842923 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.842847    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-credential-socket\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.842923 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.842892    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-workload-certs\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.843304 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.843254    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-workload-certs\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.843993 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.843916    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-istio-envoy\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.844904 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.844876    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ef253e-1870-4b85-ab78-f155131276a9-istio-podinfo\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.894975 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.894944    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:34.898172 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.898144    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ef253e-1870-4b85-ab78-f155131276a9-istio-token\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.898297 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.898208    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lts6d\" (UniqueName: \"kubernetes.io/projected/91ef253e-1870-4b85-ab78-f155131276a9-kube-api-access-lts6d\") pod \"e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") " pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:34.914443 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:34.914419    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:35.369249 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:03:35.369215    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ef253e_1870_4b85_ab78_f155131276a9.slice/crio-a53230ca5cc69583514b39fbf3ab3a5cb1dc3ab8621311a4f4885fa7dabd80e9 WatchSource:0}: Error finding container a53230ca5cc69583514b39fbf3ab3a5cb1dc3ab8621311a4f4885fa7dabd80e9: Status 404 returned error can't find the container with id a53230ca5cc69583514b39fbf3ab3a5cb1dc3ab8621311a4f4885fa7dabd80e9
Oct 30 19:03:35.371381 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:35.371356    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"]
Oct 30 19:03:35.371733 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:35.371703    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:03:35.371780 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:35.371768    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:03:35.371828 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:35.371801    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:03:35.451249 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:35.450718    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4" event={"ID":"91ef253e-1870-4b85-ab78-f155131276a9","Type":"ContainerStarted","Data":"a53230ca5cc69583514b39fbf3ab3a5cb1dc3ab8621311a4f4885fa7dabd80e9"}
Oct 30 19:03:35.482825 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:35.482775    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:36.460229 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:36.460192    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4" event={"ID":"91ef253e-1870-4b85-ab78-f155131276a9","Type":"ContainerStarted","Data":"c4d0f33f078f056566b5489b4da4e50807261f8bd73b63603ed899d11fcafdea"}
Oct 30 19:03:36.670653 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:36.670601    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4" podStartSLOduration=2.670587951 podStartE2EDuration="2.670587951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:36.620178043 +0000 UTC m=+1617.897228488" watchObservedRunningTime="2026-10-30 19:03:36.670587951 +0000 UTC m=+1617.947638396"
Oct 30 19:03:36.915463 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:36.915432    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:36.920971 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:36.920945    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:37.270336 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:37.270232    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:37.445532 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:37.445498    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:37.469136 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:37.469104    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:37.470047 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:37.470030    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:03:39.051035 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:39.050935    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:39.227306 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:39.227247    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:39.431564 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:39.431527    4924 scope.go:122] "RemoveContainer" containerID="2d5cbf2ac72e53278cf3e8912f093e40e806619e00e49c2b486a593b8d949485"
Oct 30 19:03:39.460489 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:39.460465    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:39.654437 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:39.654406    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:39.854910 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:39.854880    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:40.233170 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:40.226814    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:42.523275 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:42.523248    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:42.710423 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:42.710377    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:42.891221 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:42.890772    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr"]
Oct 30 19:03:42.915232 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:42.915197    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr"
Oct 30 19:03:42.953495 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:42.953458    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr"]
Oct 30 19:03:42.995115 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:42.990391    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-8be536\""
Oct 30 19:03:42.995115 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:42.990572    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-8be536-dockercfg-8p9kd\""
Oct 30 19:03:43.119569 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:43.119461    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc"]
Oct 30 19:03:43.126473 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:43.125690    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc"
Oct 30 19:03:43.134239 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:43.134213    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc"]
Oct 30 19:03:43.171006 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:43.170974    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5c238f-665a-4690-8148-cb92c3dd8580-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr\" (UID: \"3f5c238f-665a-4690-8148-cb92c3dd8580\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr"
Oct 30 19:03:43.171200 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:43.171029    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkdq4\" (UniqueName: \"kubernetes.io/projected/3f5c238f-665a-4690-8148-cb92c3dd8580-kube-api-access-mkdq4\") pod \"payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr\" (UID: \"3f5c238f-665a-4690-8148-cb92c3dd8580\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr"
Oct 30 19:03:43.272623 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:43.272562    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba79faa-3308-4f5a-b30a-9e569ca5d466-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc\" (UID: \"9ba79faa-3308-4f5a-b30a-9e569ca5d466\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc"
Oct 30 19:03:43.273007 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:43.272980    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5c238f-665a-4690-8148-cb92c3dd8580-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr\" (UID: \"3f5c238f-665a-4690-8148-cb92c3dd8580\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr"
Oct 30 19:03:43.274052 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:43.273159    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkdq4\" (UniqueName: \"kubernetes.io/projected/3f5c238f-665a-4690-8148-cb92c3dd8580-kube-api-access-mkdq4\") pod \"payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr\" (UID: \"3f5c238f-665a-4690-8148-cb92c3dd8580\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr"
Oct 30 19:03:43.274052 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:43.273470    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cbj5\" (UniqueName: \"kubernetes.io/projected/9ba79faa-3308-4f5a-b30a-9e569ca5d466-kube-api-access-4cbj5\") pod \"payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc\" (UID: \"9ba79faa-3308-4f5a-b30a-9e569ca5d466\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc"
Oct 30 19:03:43.274052 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:43.274004    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5c238f-665a-4690-8148-cb92c3dd8580-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr\" (UID: \"3f5c238f-665a-4690-8148-cb92c3dd8580\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr"
Oct 30 19:03:43.330773 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:43.330738    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkdq4\" (UniqueName: \"kubernetes.io/projected/3f5c238f-665a-4690-8148-cb92c3dd8580-kube-api-access-mkdq4\") pod \"payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr\" (UID: \"3f5c238f-665a-4690-8148-cb92c3dd8580\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr"
Oct 30 19:03:43.374788 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:43.374754    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cbj5\" (UniqueName: \"kubernetes.io/projected/9ba79faa-3308-4f5a-b30a-9e569ca5d466-kube-api-access-4cbj5\") pod \"payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc\" (UID: \"9ba79faa-3308-4f5a-b30a-9e569ca5d466\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc"
Oct 30 19:03:43.374986 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:43.374814    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba79faa-3308-4f5a-b30a-9e569ca5d466-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc\" (UID: \"9ba79faa-3308-4f5a-b30a-9e569ca5d466\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc"
Oct 30 19:03:43.375537 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:43.375515    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba79faa-3308-4f5a-b30a-9e569ca5d466-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc\" (UID: \"9ba79faa-3308-4f5a-b30a-9e569ca5d466\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc"
Oct 30 19:03:43.428775 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:43.428717    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cbj5\" (UniqueName: \"kubernetes.io/projected/9ba79faa-3308-4f5a-b30a-9e569ca5d466-kube-api-access-4cbj5\") pod \"payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc\" (UID: \"9ba79faa-3308-4f5a-b30a-9e569ca5d466\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc"
Oct 30 19:03:43.448919 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:43.448892    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc"
Oct 30 19:03:43.540433 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:43.540361    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr"
Oct 30 19:03:43.820263 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:03:43.820220    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba79faa_3308_4f5a_b30a_9e569ca5d466.slice/crio-9d52b33c806fda732b6e1a4c82bd3ef10e028de04231c18f5af1dc74563ab1f7 WatchSource:0}: Error finding container 9d52b33c806fda732b6e1a4c82bd3ef10e028de04231c18f5af1dc74563ab1f7: Status 404 returned error can't find the container with id 9d52b33c806fda732b6e1a4c82bd3ef10e028de04231c18f5af1dc74563ab1f7
Oct 30 19:03:43.822039 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:43.822019    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc"]
Oct 30 19:03:43.924729 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:03:43.924700    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5c238f_665a_4690_8148_cb92c3dd8580.slice/crio-c564a2a62b1a166db32e2c575a84c29c0cd37b9bb6af53f43556a94c5a2cb1a6 WatchSource:0}: Error finding container c564a2a62b1a166db32e2c575a84c29c0cd37b9bb6af53f43556a94c5a2cb1a6: Status 404 returned error can't find the container with id c564a2a62b1a166db32e2c575a84c29c0cd37b9bb6af53f43556a94c5a2cb1a6
Oct 30 19:03:43.926196 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:43.926175    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr"]
Oct 30 19:03:44.110976 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:44.110937    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:44.535316 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:44.535213    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc" event={"ID":"9ba79faa-3308-4f5a-b30a-9e569ca5d466","Type":"ContainerStarted","Data":"587edc08df4c8cd2782e089f6caf19e9b50560259c8a8c86b982966549934397"}
Oct 30 19:03:44.535486 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:44.535324    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc"
Oct 30 19:03:44.535486 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:44.535350    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc" event={"ID":"9ba79faa-3308-4f5a-b30a-9e569ca5d466","Type":"ContainerStarted","Data":"9d52b33c806fda732b6e1a4c82bd3ef10e028de04231c18f5af1dc74563ab1f7"}
Oct 30 19:03:44.537382 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:44.537347    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr" event={"ID":"3f5c238f-665a-4690-8148-cb92c3dd8580","Type":"ContainerStarted","Data":"47d2e0bd4d755e8259efeb11ad187aba122eeb3a02a0997d007849eff7adf271"}
Oct 30 19:03:44.537522 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:44.537404    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr" event={"ID":"3f5c238f-665a-4690-8148-cb92c3dd8580","Type":"ContainerStarted","Data":"c564a2a62b1a166db32e2c575a84c29c0cd37b9bb6af53f43556a94c5a2cb1a6"}
Oct 30 19:03:44.537725 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:44.537699    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr"
Oct 30 19:03:44.760574 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:44.760511    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc" podStartSLOduration=1.760493405 podStartE2EDuration="1.760493405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:44.646283264 +0000 UTC m=+1625.923333709" watchObservedRunningTime="2026-10-30 19:03:44.760493405 +0000 UTC m=+1626.037543852"
Oct 30 19:03:44.762329 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:44.762293    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr" podStartSLOduration=2.762278238 podStartE2EDuration="2.762278238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:44.754199094 +0000 UTC m=+1626.031249539" watchObservedRunningTime="2026-10-30 19:03:44.762278238 +0000 UTC m=+1626.039328687"
Oct 30 19:03:45.049582 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:45.049546    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82aee4e5-fnt28"]
Oct 30 19:03:45.054740 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:45.054711    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82aee4e5-fnt28"]
Oct 30 19:03:45.608784 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:45.608755    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:45.714276 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:03:45.714243    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:03:45.823731 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:45.823700    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:45.996556 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:45.996463    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:46.069860 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:46.069825    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-948bc488-phkkb"]
Oct 30 19:03:46.115737 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:46.115704    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-948bc488-phkkb"]
Oct 30 19:03:46.145902 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:46.145868    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"]
Oct 30 19:03:46.146143 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:46.146114    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v" podUID="82102463-ed46-40e7-8c8f-becb671573e7" containerName="istio-proxy" containerID="cri-o://2747e0cc030a9e55545cbdec7f5567b0fdca65aa57fd13f1e26c8c9f0df469af" gracePeriod=30
Oct 30 19:03:46.229184 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:46.229155    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:46.837197 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:46.837168    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bca2977-14c1-4b41-b784-6e93ab323aa5" path="/var/lib/kubelet/pods/1bca2977-14c1-4b41-b784-6e93ab323aa5/volumes"
Oct 30 19:03:46.837683 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:46.837667    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="add5882b-25a6-4736-b414-9ebea55362e0" path="/var/lib/kubelet/pods/add5882b-25a6-4736-b414-9ebea55362e0/volumes"
Oct 30 19:03:46.884528 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:46.884496    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:47.368391 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.368360    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"]
Oct 30 19:03:47.374498 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.374481    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.426747 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.426714    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-4cef683f-openshift-default-dockercfg-bjwxr\""
Oct 30 19:03:47.485077 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.485018    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-workload-socket\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.485547 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.485127    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-token\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.485713 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.485696    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-envoy\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.485785 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.485745    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqn5l\" (UniqueName: \"kubernetes.io/projected/77664ce4-7a08-4b33-bccf-2e0755f13be3-kube-api-access-tqn5l\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.485839 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.485827    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-workload-certs\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.485907 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.485852    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77664ce4-7a08-4b33-bccf-2e0755f13be3-istiod-ca-cert\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.485960 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.485915    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-podinfo\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.486009 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.485971    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-data\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.486123 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.486084    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-credential-socket\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.499533 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.499496    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"]
Oct 30 19:03:47.566114 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.561739    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:47.589444 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.589399    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-envoy\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.589607 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.589470    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqn5l\" (UniqueName: \"kubernetes.io/projected/77664ce4-7a08-4b33-bccf-2e0755f13be3-kube-api-access-tqn5l\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.589607 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.589559    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-workload-certs\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.589607 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.589585    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77664ce4-7a08-4b33-bccf-2e0755f13be3-istiod-ca-cert\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.589742 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.589646    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-podinfo\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.589742 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.589701    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-data\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.589845 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.589791    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-credential-socket\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.589922 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.589902    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-workload-socket\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.589992 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.589920    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-workload-certs\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.590049 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.590009    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-token\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.590403 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.590198    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-credential-socket\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.590499 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.590454    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77664ce4-7a08-4b33-bccf-2e0755f13be3-istiod-ca-cert\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.590622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.590393    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-workload-socket\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.590754 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.590734    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-data\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.593125 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.592793    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-envoy\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.593701 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.593677    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-podinfo\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.649532 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.647347    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-token\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.656393 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.656362    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqn5l\" (UniqueName: \"kubernetes.io/projected/77664ce4-7a08-4b33-bccf-2e0755f13be3-kube-api-access-tqn5l\") pod \"e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") " pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.690410 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.690379    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:47.975991 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:47.975913    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:48.139753 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:48.139722    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"]
Oct 30 19:03:48.168683 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:03:48.168652    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77664ce4_7a08_4b33_bccf_2e0755f13be3.slice/crio-c3480923532f4bbbb2a1fdc9b003d7b53d3f016ab18935fca5282fdaec828bbf WatchSource:0}: Error finding container c3480923532f4bbbb2a1fdc9b003d7b53d3f016ab18935fca5282fdaec828bbf: Status 404 returned error can't find the container with id c3480923532f4bbbb2a1fdc9b003d7b53d3f016ab18935fca5282fdaec828bbf
Oct 30 19:03:48.171561 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:48.171526    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:03:48.171681 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:48.171651    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:03:48.171742 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:48.171695    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:03:48.592136 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:48.592007    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn" event={"ID":"77664ce4-7a08-4b33-bccf-2e0755f13be3","Type":"ContainerStarted","Data":"fcb1e7fd8d711d7a9fbbaf24aa04a394a144ac4ca60c43d5cadbf94b24d85766"}
Oct 30 19:03:48.592136 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:48.592098    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn" event={"ID":"77664ce4-7a08-4b33-bccf-2e0755f13be3","Type":"ContainerStarted","Data":"c3480923532f4bbbb2a1fdc9b003d7b53d3f016ab18935fca5282fdaec828bbf"}
Oct 30 19:03:48.691314 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:48.691281    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:48.711360 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:48.711305    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn" podStartSLOduration=1.711288881 podStartE2EDuration="1.711288881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:48.706810771 +0000 UTC m=+1629.983861216" watchObservedRunningTime="2026-10-30 19:03:48.711288881 +0000 UTC m=+1629.988339326"
Oct 30 19:03:49.696494 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:49.696466    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:50.373739 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:50.373695    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:50.618781 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:50.618751    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:50.622514 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:50.622487    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:03:50.940573 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:50.940539    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:51.177924 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.177894    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:51.368712 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.368664    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:51.603125 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.603102    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:03:51.627299 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.627267    4924 generic.go:356] "Generic (PLEG): container finished" podID="82102463-ed46-40e7-8c8f-becb671573e7" containerID="2747e0cc030a9e55545cbdec7f5567b0fdca65aa57fd13f1e26c8c9f0df469af" exitCode=0
Oct 30 19:03:51.627488 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.627318    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"
Oct 30 19:03:51.627488 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.627354    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v" event={"ID":"82102463-ed46-40e7-8c8f-becb671573e7","Type":"ContainerDied","Data":"2747e0cc030a9e55545cbdec7f5567b0fdca65aa57fd13f1e26c8c9f0df469af"}
Oct 30 19:03:51.627488 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.627448    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v" event={"ID":"82102463-ed46-40e7-8c8f-becb671573e7","Type":"ContainerDied","Data":"e33578e9f34ade523d2583877b943971470e6ce9fb79dd5d9720ed461a9e40a2"}
Oct 30 19:03:51.627488 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.627478    4924 scope.go:122] "RemoveContainer" containerID="2747e0cc030a9e55545cbdec7f5567b0fdca65aa57fd13f1e26c8c9f0df469af"
Oct 30 19:03:51.639189 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.639161    4924 scope.go:122] "RemoveContainer" containerID="96c968c38085700c1c8e86121b29f7e5509fb9650ec2cebcac23adfd3c5d680c"
Oct 30 19:03:51.649438 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.649411    4924 scope.go:122] "RemoveContainer" containerID="2747e0cc030a9e55545cbdec7f5567b0fdca65aa57fd13f1e26c8c9f0df469af"
Oct 30 19:03:51.649718 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:03:51.649697    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2747e0cc030a9e55545cbdec7f5567b0fdca65aa57fd13f1e26c8c9f0df469af\": container with ID starting with 2747e0cc030a9e55545cbdec7f5567b0fdca65aa57fd13f1e26c8c9f0df469af not found: ID does not exist" containerID="2747e0cc030a9e55545cbdec7f5567b0fdca65aa57fd13f1e26c8c9f0df469af"
Oct 30 19:03:51.649765 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.649727    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2747e0cc030a9e55545cbdec7f5567b0fdca65aa57fd13f1e26c8c9f0df469af"} err="failed to get container status \"2747e0cc030a9e55545cbdec7f5567b0fdca65aa57fd13f1e26c8c9f0df469af\": rpc error: code = NotFound desc = could not find container \"2747e0cc030a9e55545cbdec7f5567b0fdca65aa57fd13f1e26c8c9f0df469af\": container with ID starting with 2747e0cc030a9e55545cbdec7f5567b0fdca65aa57fd13f1e26c8c9f0df469af not found: ID does not exist"
Oct 30 19:03:51.649765 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.649745    4924 scope.go:122] "RemoveContainer" containerID="96c968c38085700c1c8e86121b29f7e5509fb9650ec2cebcac23adfd3c5d680c"
Oct 30 19:03:51.650005 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:03:51.649986    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96c968c38085700c1c8e86121b29f7e5509fb9650ec2cebcac23adfd3c5d680c\": container with ID starting with 96c968c38085700c1c8e86121b29f7e5509fb9650ec2cebcac23adfd3c5d680c not found: ID does not exist" containerID="96c968c38085700c1c8e86121b29f7e5509fb9650ec2cebcac23adfd3c5d680c"
Oct 30 19:03:51.650076 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.650011    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96c968c38085700c1c8e86121b29f7e5509fb9650ec2cebcac23adfd3c5d680c"} err="failed to get container status \"96c968c38085700c1c8e86121b29f7e5509fb9650ec2cebcac23adfd3c5d680c\": rpc error: code = NotFound desc = could not find container \"96c968c38085700c1c8e86121b29f7e5509fb9650ec2cebcac23adfd3c5d680c\": container with ID starting with 96c968c38085700c1c8e86121b29f7e5509fb9650ec2cebcac23adfd3c5d680c not found: ID does not exist"
Oct 30 19:03:51.668627 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.668603    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-istio-envoy\") pod \"82102463-ed46-40e7-8c8f-becb671573e7\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") "
Oct 30 19:03:51.668767 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.668638    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82102463-ed46-40e7-8c8f-becb671573e7-kube-api-access-jpvmd\" (UniqueName: \"kubernetes.io/projected/82102463-ed46-40e7-8c8f-becb671573e7-kube-api-access-jpvmd\") pod \"82102463-ed46-40e7-8c8f-becb671573e7\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") "
Oct 30 19:03:51.668767 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.668684    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-credential-socket\") pod \"82102463-ed46-40e7-8c8f-becb671573e7\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") "
Oct 30 19:03:51.668767 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.668723    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82102463-ed46-40e7-8c8f-becb671573e7-istio-token\" (UniqueName: \"kubernetes.io/projected/82102463-ed46-40e7-8c8f-becb671573e7-istio-token\") pod \"82102463-ed46-40e7-8c8f-becb671573e7\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") "
Oct 30 19:03:51.668873 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.668818    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-workload-socket\") pod \"82102463-ed46-40e7-8c8f-becb671573e7\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") "
Oct 30 19:03:51.668946 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.668918    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82102463-ed46-40e7-8c8f-becb671573e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82102463-ed46-40e7-8c8f-becb671573e7-istiod-ca-cert\") pod \"82102463-ed46-40e7-8c8f-becb671573e7\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") "
Oct 30 19:03:51.669009 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.668992    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82102463-ed46-40e7-8c8f-becb671573e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82102463-ed46-40e7-8c8f-becb671573e7-istio-podinfo\") pod \"82102463-ed46-40e7-8c8f-becb671573e7\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") "
Oct 30 19:03:51.669009 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.668991    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-credential-socket" pod "82102463-ed46-40e7-8c8f-becb671573e7" (UID: "82102463-ed46-40e7-8c8f-becb671573e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:51.669140 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.669111    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-workload-certs\") pod \"82102463-ed46-40e7-8c8f-becb671573e7\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") "
Oct 30 19:03:51.669193 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.669174    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-istio-data\") pod \"82102463-ed46-40e7-8c8f-becb671573e7\" (UID: \"82102463-ed46-40e7-8c8f-becb671573e7\") "
Oct 30 19:03:51.669253 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.669184    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-workload-socket" pod "82102463-ed46-40e7-8c8f-becb671573e7" (UID: "82102463-ed46-40e7-8c8f-becb671573e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:51.669458 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.669395    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-workload-certs" pod "82102463-ed46-40e7-8c8f-becb671573e7" (UID: "82102463-ed46-40e7-8c8f-becb671573e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:51.669458 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.669404    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82102463-ed46-40e7-8c8f-becb671573e7-istiod-ca-cert" pod "82102463-ed46-40e7-8c8f-becb671573e7" (UID: "82102463-ed46-40e7-8c8f-becb671573e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:03:51.669663 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.669642    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-istio-data" pod "82102463-ed46-40e7-8c8f-becb671573e7" (UID: "82102463-ed46-40e7-8c8f-becb671573e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:51.670190 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.670159    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:51.670190 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.670185    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:51.670330 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.670200    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82102463-ed46-40e7-8c8f-becb671573e7-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:51.670330 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.670217    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:51.670330 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.670231    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:51.671108 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.671060    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-istio-envoy" pod "82102463-ed46-40e7-8c8f-becb671573e7" (UID: "82102463-ed46-40e7-8c8f-becb671573e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:51.671337 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.671317    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82102463-ed46-40e7-8c8f-becb671573e7-istio-token" pod "82102463-ed46-40e7-8c8f-becb671573e7" (UID: "82102463-ed46-40e7-8c8f-becb671573e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:03:51.671444 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.671427    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82102463-ed46-40e7-8c8f-becb671573e7-istio-podinfo" pod "82102463-ed46-40e7-8c8f-becb671573e7" (UID: "82102463-ed46-40e7-8c8f-becb671573e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:03:51.671495 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.671474    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82102463-ed46-40e7-8c8f-becb671573e7-kube-api-access-jpvmd" pod "82102463-ed46-40e7-8c8f-becb671573e7" (UID: "82102463-ed46-40e7-8c8f-becb671573e7"). InnerVolumeSpecName "kube-api-access-jpvmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:03:51.772505 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.772468    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82102463-ed46-40e7-8c8f-becb671573e7-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:51.772505 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.772508    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpvmd\" (UniqueName: \"kubernetes.io/projected/82102463-ed46-40e7-8c8f-becb671573e7-kube-api-access-jpvmd\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:51.772707 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.772524    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82102463-ed46-40e7-8c8f-becb671573e7-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:51.772707 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:51.772537    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82102463-ed46-40e7-8c8f-becb671573e7-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:52.011783 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:52.011747    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:52.049016 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:52.048984    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"]
Oct 30 19:03:52.054141 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:52.054118    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-948bc488-gw-openshift-default-55dc45d749-c277v"]
Oct 30 19:03:52.206428 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:52.206330    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:52.395448 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:52.395027    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:52.495475 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:52.495387    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n"]
Oct 30 19:03:52.500400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:52.500372    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85b6ee7b-bd96n"]
Oct 30 19:03:52.836495 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:52.836409    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42b85dc2-97ed-4a94-b7ad-4b2d45635e10" path="/var/lib/kubelet/pods/42b85dc2-97ed-4a94-b7ad-4b2d45635e10/volumes"
Oct 30 19:03:52.837043 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:52.837019    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82102463-ed46-40e7-8c8f-becb671573e7" path="/var/lib/kubelet/pods/82102463-ed46-40e7-8c8f-becb671573e7/volumes"
Oct 30 19:03:54.012942 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:54.012913    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"]
Oct 30 19:03:54.013370 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:54.013136    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf" podUID="4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f" containerName="istio-proxy" containerID="cri-o://8897ef6c9480c61746d8647531758504ea98ac0120d0e3b40e6fcc8b0d87c553" gracePeriod=30
Oct 30 19:03:54.058984 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:54.058944    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:54.587986 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:54.587947    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:54.779651 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:54.779617    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:55.073950 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:55.073915    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:55.268219 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:55.268187    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:55.550982 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:55.550953    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc"
Oct 30 19:03:55.551201 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:55.551033    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr"
Oct 30 19:03:56.056181 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:56.056150    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5eaa0bd-lrwlw"]
Oct 30 19:03:56.061477 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:56.061450    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5eaa0bd-lrwlw"]
Oct 30 19:03:56.423798 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:56.423772    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:56.597294 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:56.597265    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:56.836922 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:56.836844    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88059f5b-33a7-407d-8c56-5656fc90cace" path="/var/lib/kubelet/pods/88059f5b-33a7-407d-8c56-5656fc90cace/volumes"
Oct 30 19:03:57.262602 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:57.262558    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8040-txg9n"]
Oct 30 19:03:57.263406 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:57.263386    4924 cpu_manager.go:414] "containerMap: removing container" podUID="82102463-ed46-40e7-8c8f-becb671573e7" containerName="istio-proxy"
Oct 30 19:03:57.263468 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:57.263408    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82102463-ed46-40e7-8c8f-becb671573e7" containerName="istio-proxy"
Oct 30 19:03:57.263468 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:57.263420    4924 cpu_manager.go:414] "containerMap: removing container" podUID="82102463-ed46-40e7-8c8f-becb671573e7" containerName="istio-proxy"
Oct 30 19:03:57.263468 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:57.263425    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82102463-ed46-40e7-8c8f-becb671573e7" containerName="istio-proxy"
Oct 30 19:03:57.263580 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:57.263567    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="82102463-ed46-40e7-8c8f-becb671573e7" containerName="istio-proxy"
Oct 30 19:03:57.263615 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:57.263583    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="82102463-ed46-40e7-8c8f-becb671573e7" containerName="istio-proxy"
Oct 30 19:03:57.270213 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:57.270183    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8040-txg9n"
Oct 30 19:03:57.321415 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:57.321379    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8040-txg9n"]
Oct 30 19:03:57.358939 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:57.358904    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:03:57.359218 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:57.358950    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:03:57.365251 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:57.365224    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/001c4e0f-57d2-4624-8c64-2f5afaaabeba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba3a8040-txg9n\" (UID: \"001c4e0f-57d2-4624-8c64-2f5afaaabeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8040-txg9n"
Oct 30 19:03:57.365403 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:57.365293    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/001c4e0f-57d2-4624-8c64-2f5afaaabeba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba3a8040-txg9n\" (UID: \"001c4e0f-57d2-4624-8c64-2f5afaaabeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8040-txg9n"
Oct 30 19:03:57.365546 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:57.365524    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkzdp\" (UniqueName: \"kubernetes.io/projected/001c4e0f-57d2-4624-8c64-2f5afaaabeba-kube-api-access-qkzdp\") pod \"maas-api-revoke-sub-ba3a8040-txg9n\" (UID: \"001c4e0f-57d2-4624-8c64-2f5afaaabeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8040-txg9n"
Oct 30 19:03:57.365848 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:57.365827    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:03:57.365969 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:57.365868    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:03:57.467679 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:57.467643    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/001c4e0f-57d2-4624-8c64-2f5afaaabeba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba3a8040-txg9n\" (UID: \"001c4e0f-57d2-4624-8c64-2f5afaaabeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8040-txg9n"
Oct 30 19:03:57.468128 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:57.467737    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/001c4e0f-57d2-4624-8c64-2f5afaaabeba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba3a8040-txg9n\" (UID: \"001c4e0f-57d2-4624-8c64-2f5afaaabeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8040-txg9n"
Oct 30 19:03:57.468128 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:57.467891    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkzdp\" (UniqueName: \"kubernetes.io/projected/001c4e0f-57d2-4624-8c64-2f5afaaabeba-kube-api-access-qkzdp\") pod \"maas-api-revoke-sub-ba3a8040-txg9n\" (UID: \"001c4e0f-57d2-4624-8c64-2f5afaaabeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8040-txg9n"
Oct 30 19:03:57.468474 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:57.468456    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/001c4e0f-57d2-4624-8c64-2f5afaaabeba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba3a8040-txg9n\" (UID: \"001c4e0f-57d2-4624-8c64-2f5afaaabeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8040-txg9n"
Oct 30 19:03:57.470151 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:57.470125    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/001c4e0f-57d2-4624-8c64-2f5afaaabeba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba3a8040-txg9n\" (UID: \"001c4e0f-57d2-4624-8c64-2f5afaaabeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8040-txg9n"
Oct 30 19:03:57.522350 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:57.522285    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkzdp\" (UniqueName: \"kubernetes.io/projected/001c4e0f-57d2-4624-8c64-2f5afaaabeba-kube-api-access-qkzdp\") pod \"maas-api-revoke-sub-ba3a8040-txg9n\" (UID: \"001c4e0f-57d2-4624-8c64-2f5afaaabeba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8040-txg9n"
Oct 30 19:03:57.556307 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:57.556262    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:03:57.590759 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:57.590724    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8040-txg9n"
Oct 30 19:03:57.992818 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:57.992785    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8040-txg9n"]
Oct 30 19:03:57.996928 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:03:57.996902    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod001c4e0f_57d2_4624_8c64_2f5afaaabeba.slice/crio-2965cc6c54b384034461626c825bd81ab6939fdfd16b3f84dba3131ae03b66d1 WatchSource:0}: Error finding container 2965cc6c54b384034461626c825bd81ab6939fdfd16b3f84dba3131ae03b66d1: Status 404 returned error can't find the container with id 2965cc6c54b384034461626c825bd81ab6939fdfd16b3f84dba3131ae03b66d1
Oct 30 19:03:58.739451 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:58.739412    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8040-txg9n" event={"ID":"001c4e0f-57d2-4624-8c64-2f5afaaabeba","Type":"ContainerStarted","Data":"d3c8055238d6e4012b85b8c35fc3575ce8b9436b199007ea94cca9e42868faa9"}
Oct 30 19:03:58.739823 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:58.739470    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8040-txg9n" event={"ID":"001c4e0f-57d2-4624-8c64-2f5afaaabeba","Type":"ContainerStarted","Data":"2965cc6c54b384034461626c825bd81ab6939fdfd16b3f84dba3131ae03b66d1"}
Oct 30 19:03:58.851807 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:58.851744    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8040-txg9n" podStartSLOduration=1.851730544 podStartE2EDuration="1.851730544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:58.847003824 +0000 UTC m=+1640.124054268" watchObservedRunningTime="2026-10-30 19:03:58.851730544 +0000 UTC m=+1640.128780989"
Oct 30 19:03:59.295628 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.295574    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:03:59.402187 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.402157    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-workload-socket\") pod \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") "
Oct 30 19:03:59.402338 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.402223    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-kube-api-access-wsjnz\" (UniqueName: \"kubernetes.io/projected/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-kube-api-access-wsjnz\") pod \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") "
Oct 30 19:03:59.402338 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.402283    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-data\") pod \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") "
Oct 30 19:03:59.402431 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.402355    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-envoy\") pod \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") "
Oct 30 19:03:59.402431 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.402408    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istiod-ca-cert\") pod \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") "
Oct 30 19:03:59.402538 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.402434    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-credential-socket\") pod \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") "
Oct 30 19:03:59.402538 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.402442    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-workload-socket" pod "4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f" (UID: "4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:59.402538 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.402470    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-token\" (UniqueName: \"kubernetes.io/projected/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-token\") pod \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") "
Oct 30 19:03:59.402701 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.402562    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-data" pod "4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f" (UID: "4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:59.402701 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.402583    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-workload-certs\") pod \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") "
Oct 30 19:03:59.402842 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.402721    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-podinfo\") pod \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\" (UID: \"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f\") "
Oct 30 19:03:59.402842 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.402763    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-workload-certs" pod "4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f" (UID: "4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:59.402842 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.402785    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-credential-socket" pod "4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f" (UID: "4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:59.403015 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.402845    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istiod-ca-cert" pod "4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f" (UID: "4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:03:59.403824 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.403797    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:59.403944 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.403828    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:59.403944 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.403844    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:59.403944 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.403859    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:59.403944 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.403874    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:59.404697 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.404672    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-kube-api-access-wsjnz" pod "4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f" (UID: "4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f"). InnerVolumeSpecName "kube-api-access-wsjnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:03:59.404773 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.404732    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-envoy" pod "4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f" (UID: "4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:59.404821 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.404774    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-podinfo" pod "4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f" (UID: "4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:03:59.404868 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.404851    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-token" pod "4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f" (UID: "4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:03:59.505949 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.505910    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:59.505949 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.505944    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsjnz\" (UniqueName: \"kubernetes.io/projected/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-kube-api-access-wsjnz\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:59.506161 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.505960    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:59.506161 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.505972    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:59.756693 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.756657    4924 generic.go:356] "Generic (PLEG): container finished" podID="4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f" containerID="8897ef6c9480c61746d8647531758504ea98ac0120d0e3b40e6fcc8b0d87c553" exitCode=0
Oct 30 19:03:59.757212 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.756711    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"
Oct 30 19:03:59.757212 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.756740    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf" event={"ID":"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f","Type":"ContainerDied","Data":"8897ef6c9480c61746d8647531758504ea98ac0120d0e3b40e6fcc8b0d87c553"}
Oct 30 19:03:59.757212 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.756818    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf" event={"ID":"4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f","Type":"ContainerDied","Data":"bc1390f82ea4a510fee4ac636b96e288482a25141b9380fa5435aa8a87ba3781"}
Oct 30 19:03:59.757212 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.756856    4924 scope.go:122] "RemoveContainer" containerID="8897ef6c9480c61746d8647531758504ea98ac0120d0e3b40e6fcc8b0d87c553"
Oct 30 19:03:59.758986 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.758965    4924 generic.go:356] "Generic (PLEG): container finished" podID="001c4e0f-57d2-4624-8c64-2f5afaaabeba" containerID="d3c8055238d6e4012b85b8c35fc3575ce8b9436b199007ea94cca9e42868faa9" exitCode=0
Oct 30 19:03:59.759133 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.759041    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8040-txg9n" event={"ID":"001c4e0f-57d2-4624-8c64-2f5afaaabeba","Type":"ContainerDied","Data":"d3c8055238d6e4012b85b8c35fc3575ce8b9436b199007ea94cca9e42868faa9"}
Oct 30 19:03:59.770207 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.770183    4924 scope.go:122] "RemoveContainer" containerID="0aaafcbd95cecf4b847201cfdf0d4c93a0a5f69b040770392c0d9b184a7ce0d9"
Oct 30 19:03:59.780461 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.780440    4924 scope.go:122] "RemoveContainer" containerID="8897ef6c9480c61746d8647531758504ea98ac0120d0e3b40e6fcc8b0d87c553"
Oct 30 19:03:59.780733 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:03:59.780715    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8897ef6c9480c61746d8647531758504ea98ac0120d0e3b40e6fcc8b0d87c553\": container with ID starting with 8897ef6c9480c61746d8647531758504ea98ac0120d0e3b40e6fcc8b0d87c553 not found: ID does not exist" containerID="8897ef6c9480c61746d8647531758504ea98ac0120d0e3b40e6fcc8b0d87c553"
Oct 30 19:03:59.780799 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.780741    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8897ef6c9480c61746d8647531758504ea98ac0120d0e3b40e6fcc8b0d87c553"} err="failed to get container status \"8897ef6c9480c61746d8647531758504ea98ac0120d0e3b40e6fcc8b0d87c553\": rpc error: code = NotFound desc = could not find container \"8897ef6c9480c61746d8647531758504ea98ac0120d0e3b40e6fcc8b0d87c553\": container with ID starting with 8897ef6c9480c61746d8647531758504ea98ac0120d0e3b40e6fcc8b0d87c553 not found: ID does not exist"
Oct 30 19:03:59.780799 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.780757    4924 scope.go:122] "RemoveContainer" containerID="0aaafcbd95cecf4b847201cfdf0d4c93a0a5f69b040770392c0d9b184a7ce0d9"
Oct 30 19:03:59.780977 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:03:59.780961    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aaafcbd95cecf4b847201cfdf0d4c93a0a5f69b040770392c0d9b184a7ce0d9\": container with ID starting with 0aaafcbd95cecf4b847201cfdf0d4c93a0a5f69b040770392c0d9b184a7ce0d9 not found: ID does not exist" containerID="0aaafcbd95cecf4b847201cfdf0d4c93a0a5f69b040770392c0d9b184a7ce0d9"
Oct 30 19:03:59.781018 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.780981    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aaafcbd95cecf4b847201cfdf0d4c93a0a5f69b040770392c0d9b184a7ce0d9"} err="failed to get container status \"0aaafcbd95cecf4b847201cfdf0d4c93a0a5f69b040770392c0d9b184a7ce0d9\": rpc error: code = NotFound desc = could not find container \"0aaafcbd95cecf4b847201cfdf0d4c93a0a5f69b040770392c0d9b184a7ce0d9\": container with ID starting with 0aaafcbd95cecf4b847201cfdf0d4c93a0a5f69b040770392c0d9b184a7ce0d9 not found: ID does not exist"
Oct 30 19:03:59.917280 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.917251    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"]
Oct 30 19:03:59.922188 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:03:59.922168    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-85b6ee7b-openshift-default-85c67896f7-pzngf"]
Oct 30 19:04:00.845659 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:00.845454    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f" path="/var/lib/kubelet/pods/4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f/volumes"
Oct 30 19:04:00.947703 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:00.947676    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8040-txg9n"
Oct 30 19:04:01.028595 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:01.028520    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/001c4e0f-57d2-4624-8c64-2f5afaaabeba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/001c4e0f-57d2-4624-8c64-2f5afaaabeba-maas-api-xks-ca\") pod \"001c4e0f-57d2-4624-8c64-2f5afaaabeba\" (UID: \"001c4e0f-57d2-4624-8c64-2f5afaaabeba\") "
Oct 30 19:04:01.028743 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:01.028606    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/001c4e0f-57d2-4624-8c64-2f5afaaabeba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/001c4e0f-57d2-4624-8c64-2f5afaaabeba-maas-api-openshift-ca\") pod \"001c4e0f-57d2-4624-8c64-2f5afaaabeba\" (UID: \"001c4e0f-57d2-4624-8c64-2f5afaaabeba\") "
Oct 30 19:04:01.028836 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:01.028818    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/001c4e0f-57d2-4624-8c64-2f5afaaabeba-kube-api-access-qkzdp\" (UniqueName: \"kubernetes.io/projected/001c4e0f-57d2-4624-8c64-2f5afaaabeba-kube-api-access-qkzdp\") pod \"001c4e0f-57d2-4624-8c64-2f5afaaabeba\" (UID: \"001c4e0f-57d2-4624-8c64-2f5afaaabeba\") "
Oct 30 19:04:01.028982 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:01.028957    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/001c4e0f-57d2-4624-8c64-2f5afaaabeba-maas-api-openshift-ca" pod "001c4e0f-57d2-4624-8c64-2f5afaaabeba" (UID: "001c4e0f-57d2-4624-8c64-2f5afaaabeba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:01.029371 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:01.029350    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/001c4e0f-57d2-4624-8c64-2f5afaaabeba-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:01.030710 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:01.030687    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/001c4e0f-57d2-4624-8c64-2f5afaaabeba-maas-api-xks-ca" pod "001c4e0f-57d2-4624-8c64-2f5afaaabeba" (UID: "001c4e0f-57d2-4624-8c64-2f5afaaabeba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:04:01.030855 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:01.030839    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/001c4e0f-57d2-4624-8c64-2f5afaaabeba-kube-api-access-qkzdp" pod "001c4e0f-57d2-4624-8c64-2f5afaaabeba" (UID: "001c4e0f-57d2-4624-8c64-2f5afaaabeba"). InnerVolumeSpecName "kube-api-access-qkzdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:01.131189 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:01.131148    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/001c4e0f-57d2-4624-8c64-2f5afaaabeba-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:01.131189 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:01.131190    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkzdp\" (UniqueName: \"kubernetes.io/projected/001c4e0f-57d2-4624-8c64-2f5afaaabeba-kube-api-access-qkzdp\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:01.779536 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:01.779503    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8040-txg9n"
Oct 30 19:04:01.779738 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:01.779533    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8040-txg9n" event={"ID":"001c4e0f-57d2-4624-8c64-2f5afaaabeba","Type":"ContainerDied","Data":"2965cc6c54b384034461626c825bd81ab6939fdfd16b3f84dba3131ae03b66d1"}
Oct 30 19:04:01.779738 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:01.779610    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2965cc6c54b384034461626c825bd81ab6939fdfd16b3f84dba3131ae03b66d1"
Oct 30 19:04:02.746807 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:02.746770    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:03.050673 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:03.050596    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5486fdf2-2wmdg"]
Oct 30 19:04:03.097350 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:03.097323    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5486fdf2-2wmdg"]
Oct 30 19:04:04.885872 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:04.885843    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da83659c-89d9-4f6b-8d61-640f7246cb87" path="/var/lib/kubelet/pods/da83659c-89d9-4f6b-8d61-640f7246cb87/volumes"
Oct 30 19:04:05.987795 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:05.987762    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"]
Oct 30 19:04:05.988698 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:05.988674    4924 cpu_manager.go:414] "containerMap: removing container" podUID="4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f" containerName="istio-proxy"
Oct 30 19:04:05.988698 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:05.988699    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f" containerName="istio-proxy"
Oct 30 19:04:05.988865 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:05.988773    4924 cpu_manager.go:414] "containerMap: removing container" podUID="4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f" containerName="istio-proxy"
Oct 30 19:04:05.988865 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:05.988782    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f" containerName="istio-proxy"
Oct 30 19:04:05.988865 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:05.988797    4924 cpu_manager.go:414] "containerMap: removing container" podUID="001c4e0f-57d2-4624-8c64-2f5afaaabeba" containerName="revoke-keys"
Oct 30 19:04:05.988865 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:05.988804    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="001c4e0f-57d2-4624-8c64-2f5afaaabeba" containerName="revoke-keys"
Oct 30 19:04:05.989090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:05.988997    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f" containerName="istio-proxy"
Oct 30 19:04:05.989090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:05.989012    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="001c4e0f-57d2-4624-8c64-2f5afaaabeba" containerName="revoke-keys"
Oct 30 19:04:05.994235 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:05.994213    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:04:06.062311 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.062287    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 19:04:06.069896 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.069871    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7t6jq\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:04:06.070059 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.070037    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4csxk\" (UniqueName: \"kubernetes.io/projected/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-kube-api-access-4csxk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7t6jq\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:04:06.070252 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.070235    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7t6jq\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:04:06.070334 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.070264    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7t6jq\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:04:06.070334 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.070297    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7t6jq\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:04:06.070414 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.070398    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7t6jq\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:04:06.070465 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.070449    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7t6jq\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:04:06.093928 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.093903    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 19:04:06.093928 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.093913    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 19:04:06.094110 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.093942    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ntm7n\""
Oct 30 19:04:06.100471 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.100446    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"]
Oct 30 19:04:06.172185 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.172150    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7t6jq\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:04:06.172380 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.172239    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7t6jq\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:04:06.172380 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.172316    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7t6jq\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:04:06.172514 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.172448    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4csxk\" (UniqueName: \"kubernetes.io/projected/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-kube-api-access-4csxk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7t6jq\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:04:06.172635 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.172613    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7t6jq\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:04:06.172703 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.172654    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7t6jq\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:04:06.172703 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.172688    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7t6jq\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:04:06.173868 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.173799    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7t6jq\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:04:06.175479 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.175436    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7t6jq\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:04:06.176482 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.176415    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7t6jq\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:04:06.179007 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.178874    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7t6jq\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:04:06.181033 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.180992    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7t6jq\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:04:06.230192 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.230103    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4csxk\" (UniqueName: \"kubernetes.io/projected/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-kube-api-access-4csxk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7t6jq\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:04:06.230572 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.230549    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7t6jq\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:04:06.332116 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.332003    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:04:06.757707 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:04:06.757675    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1706cc50_6d51_4e2a_8cf1_308f6e2187b0.slice/crio-9ae9fb2eaff5e923317709b1fe22c7e7b76f250843c2b4e989592ba166e31b9c WatchSource:0}: Error finding container 9ae9fb2eaff5e923317709b1fe22c7e7b76f250843c2b4e989592ba166e31b9c: Status 404 returned error can't find the container with id 9ae9fb2eaff5e923317709b1fe22c7e7b76f250843c2b4e989592ba166e31b9c
Oct 30 19:04:06.758610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.758583    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"]
Oct 30 19:04:06.760161 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.760135    4924 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:04:06.930120 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:06.929620    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq" event={"ID":"1706cc50-6d51-4e2a-8cf1-308f6e2187b0","Type":"ContainerStarted","Data":"9ae9fb2eaff5e923317709b1fe22c7e7b76f250843c2b4e989592ba166e31b9c"}
Oct 30 19:04:07.562439 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:07.562380    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:07.951282 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:07.951248    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:08.049318 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:08.049282    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db6f995c-4s4kh"]
Oct 30 19:04:08.096029 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:08.096001    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db6f995c-4s4kh"]
Oct 30 19:04:08.840962 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:08.840926    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b61eb9a8-e0da-4a58-97ae-3c96292a14ec" path="/var/lib/kubelet/pods/b61eb9a8-e0da-4a58-97ae-3c96292a14ec/volumes"
Oct 30 19:04:10.907638 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:10.907592    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:11.204268 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:11.204196    4924 patch_prober.go:29] interesting pod/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:04:11.204671 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:11.204275    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9" podUID="61957487-4e2c-47d7-9977-1d34f3c2534f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:04:11.231499 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:11.231462    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:04:11.231639 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:11.231540    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:04:11.992739 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:11.992694    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq" event={"ID":"1706cc50-6d51-4e2a-8cf1-308f6e2187b0","Type":"ContainerStarted","Data":"090740e26b8a8fcab79cd332873d6dc800f14c33bf5f7ec93340edf1a5fa453a"}
Oct 30 19:04:11.994137 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:11.994094    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:04:12.189997 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:12.189960    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:12.366442 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:12.366387    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:12.683834 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:12.683697    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:13.007733 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:13.007642    4924 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-7t6jq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:04:13.007733 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:13.007692    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq" podUID="1706cc50-6d51-4e2a-8cf1-308f6e2187b0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:04:13.054330 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:13.054266    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq" podStartSLOduration=3.584667648 podStartE2EDuration="8.054247194s" totalImagesPullingTime="4.469579546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:12.206333707 +0000 UTC m=+1653.483384152" watchObservedRunningTime="2026-10-30 19:04:13.054247194 +0000 UTC m=+1654.331297640"
Oct 30 19:04:13.054514 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:13.054495    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8779c64-8klw8"]
Oct 30 19:04:13.101465 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:13.101421    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8779c64-8klw8"]
Oct 30 19:04:14.016265 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:14.016225    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:04:14.396478 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:14.396428    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:14.837314 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:14.837238    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a38942b6-7d30-4535-8be2-94542d7c8776" path="/var/lib/kubelet/pods/a38942b6-7d30-4535-8be2-94542d7c8776/volumes"
Oct 30 19:04:15.025618 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:15.025589    4924 generic.go:356] "Generic (PLEG): container finished" podID="61957487-4e2c-47d7-9977-1d34f3c2534f" containerID="24cf43daf4b82557e9a952375329352223416944d7697fc9e9ea03026a2c27ea" exitCode=0
Oct 30 19:04:15.026056 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:15.025664    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9" event={"ID":"61957487-4e2c-47d7-9977-1d34f3c2534f","Type":"ContainerDied","Data":"24cf43daf4b82557e9a952375329352223416944d7697fc9e9ea03026a2c27ea"}
Oct 30 19:04:15.026209 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:15.026192    4924 scope.go:122] "RemoveContainer" containerID="24cf43daf4b82557e9a952375329352223416944d7697fc9e9ea03026a2c27ea"
Oct 30 19:04:15.027802 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:15.027777    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:04:15.027883 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:15.027845    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:04:15.027926 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:15.027884    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:04:15.735400 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:04:15.735368    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:04:16.037027 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:16.036944    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9" event={"ID":"61957487-4e2c-47d7-9977-1d34f3c2534f","Type":"ContainerStarted","Data":"ada40b7f7abe844ddb41950a27d93d871473b4ed9aac378e4cc8cb1730245b9a"}
Oct 30 19:04:16.140915 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:16.140883    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:04:16.141139 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:16.141024    4924 patch_prober.go:29] interesting pod/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:04:16.141139 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:16.141092    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9" podUID="61957487-4e2c-47d7-9977-1d34f3c2534f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 19:04:16.553219 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:16.553190    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:17.141609 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:17.141566    4924 patch_prober.go:29] interesting pod/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:04:17.142178 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:17.141623    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9" podUID="61957487-4e2c-47d7-9977-1d34f3c2534f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 19:04:18.141710 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:18.141672    4924 patch_prober.go:29] interesting pod/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:04:18.142266 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:18.141732    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9" podUID="61957487-4e2c-47d7-9977-1d34f3c2534f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 19:04:18.183096 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:18.183025    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:18.485245 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:18.485151    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:18.666765 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:18.666719    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9"]
Oct 30 19:04:18.667856 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:18.667812    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:18.735703 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:18.735505    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15046cf2-rdfh9"]
Oct 30 19:04:18.850220 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:18.850184    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7be7cc1-a8ba-4176-8aeb-e337d524e43a" path="/var/lib/kubelet/pods/b7be7cc1-a8ba-4176-8aeb-e337d524e43a/volumes"
Oct 30 19:04:18.851191 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:18.851163    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:19.145118 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:19.143499    4924 patch_prober.go:29] interesting pod/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:04:19.145118 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:19.143565    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9" podUID="61957487-4e2c-47d7-9977-1d34f3c2534f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 19:04:19.776559 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:19.776478    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:19.969941 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:19.969903    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:20.141306 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:20.141272    4924 patch_prober.go:29] interesting pod/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:04:20.141468 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:20.141338    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9" podUID="61957487-4e2c-47d7-9977-1d34f3c2534f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 19:04:21.093487 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:21.093455    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:21.140696 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:21.140663    4924 patch_prober.go:29] interesting pod/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body=
Oct 30 19:04:21.140891 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:21.140714    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9" podUID="61957487-4e2c-47d7-9977-1d34f3c2534f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused"
Oct 30 19:04:21.333212 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:21.333180    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:22.145915 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:22.145884    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:04:22.146393 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:22.146349    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:04:22.147119 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:22.147100    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:04:23.402154 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:23.402119    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fa834f9-bwffr"]
Oct 30 19:04:23.403037 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:23.403019    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a56e5b5-78bf-4b8a-8588-6ede4c1aa62f" containerName="istio-proxy"
Oct 30 19:04:23.410952 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:23.410933    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa834f9-bwffr"
Oct 30 19:04:23.460324 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:23.460293    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fa834f9-bwffr"]
Oct 30 19:04:23.499827 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:23.499797    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:04:23.499827 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:23.499817    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:04:23.499992 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:23.499842    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:04:23.506710 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:23.506690    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:04:23.538943 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:23.538911    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ktb2\" (UniqueName: \"kubernetes.io/projected/48e2de76-33fe-4942-ac44-5ea1d92d8bde-kube-api-access-2ktb2\") pod \"maas-api-revoke-sub-8fa834f9-bwffr\" (UID: \"48e2de76-33fe-4942-ac44-5ea1d92d8bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa834f9-bwffr"
Oct 30 19:04:23.539060 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:23.538988    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48e2de76-33fe-4942-ac44-5ea1d92d8bde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fa834f9-bwffr\" (UID: \"48e2de76-33fe-4942-ac44-5ea1d92d8bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa834f9-bwffr"
Oct 30 19:04:23.539232 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:23.539207    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48e2de76-33fe-4942-ac44-5ea1d92d8bde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fa834f9-bwffr\" (UID: \"48e2de76-33fe-4942-ac44-5ea1d92d8bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa834f9-bwffr"
Oct 30 19:04:23.640599 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:23.640568    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ktb2\" (UniqueName: \"kubernetes.io/projected/48e2de76-33fe-4942-ac44-5ea1d92d8bde-kube-api-access-2ktb2\") pod \"maas-api-revoke-sub-8fa834f9-bwffr\" (UID: \"48e2de76-33fe-4942-ac44-5ea1d92d8bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa834f9-bwffr"
Oct 30 19:04:23.640812 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:23.640642    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48e2de76-33fe-4942-ac44-5ea1d92d8bde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fa834f9-bwffr\" (UID: \"48e2de76-33fe-4942-ac44-5ea1d92d8bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa834f9-bwffr"
Oct 30 19:04:23.640812 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:23.640757    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48e2de76-33fe-4942-ac44-5ea1d92d8bde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fa834f9-bwffr\" (UID: \"48e2de76-33fe-4942-ac44-5ea1d92d8bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa834f9-bwffr"
Oct 30 19:04:23.641337 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:23.641320    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48e2de76-33fe-4942-ac44-5ea1d92d8bde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fa834f9-bwffr\" (UID: \"48e2de76-33fe-4942-ac44-5ea1d92d8bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa834f9-bwffr"
Oct 30 19:04:23.643023 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:23.643004    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48e2de76-33fe-4942-ac44-5ea1d92d8bde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fa834f9-bwffr\" (UID: \"48e2de76-33fe-4942-ac44-5ea1d92d8bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa834f9-bwffr"
Oct 30 19:04:23.655600 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:23.655536    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:23.695703 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:23.695672    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ktb2\" (UniqueName: \"kubernetes.io/projected/48e2de76-33fe-4942-ac44-5ea1d92d8bde-kube-api-access-2ktb2\") pod \"maas-api-revoke-sub-8fa834f9-bwffr\" (UID: \"48e2de76-33fe-4942-ac44-5ea1d92d8bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa834f9-bwffr"
Oct 30 19:04:23.725238 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:23.725211    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa834f9-bwffr"
Oct 30 19:04:24.115784 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:04:24.115752    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e2de76_33fe_4942_ac44_5ea1d92d8bde.slice/crio-1a4e8f025de0fb3efb199dab8f15a2c96aab590368f9937a27515cdf7c8f62ed WatchSource:0}: Error finding container 1a4e8f025de0fb3efb199dab8f15a2c96aab590368f9937a27515cdf7c8f62ed: Status 404 returned error can't find the container with id 1a4e8f025de0fb3efb199dab8f15a2c96aab590368f9937a27515cdf7c8f62ed
Oct 30 19:04:24.116134 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:24.116110    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fa834f9-bwffr"]
Oct 30 19:04:24.171307 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:24.171276    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa834f9-bwffr" event={"ID":"48e2de76-33fe-4942-ac44-5ea1d92d8bde","Type":"ContainerStarted","Data":"1a4e8f025de0fb3efb199dab8f15a2c96aab590368f9937a27515cdf7c8f62ed"}
Oct 30 19:04:25.004154 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:25.004118    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:25.181548 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:25.181516    4924 generic.go:356] "Generic (PLEG): container finished" podID="48e2de76-33fe-4942-ac44-5ea1d92d8bde" containerID="ab08e90d3071dded6a06178abfdb5c2d8a262973175b48f3aee4cf4eb8bc2c9a" exitCode=0
Oct 30 19:04:25.181721 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:25.181602    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa834f9-bwffr" event={"ID":"48e2de76-33fe-4942-ac44-5ea1d92d8bde","Type":"ContainerDied","Data":"ab08e90d3071dded6a06178abfdb5c2d8a262973175b48f3aee4cf4eb8bc2c9a"}
Oct 30 19:04:25.897352 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:25.897301    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:26.110225 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:26.110193    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:26.379514 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:26.379485    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa834f9-bwffr"
Oct 30 19:04:26.486954 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:26.486863    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48e2de76-33fe-4942-ac44-5ea1d92d8bde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48e2de76-33fe-4942-ac44-5ea1d92d8bde-maas-api-xks-ca\") pod \"48e2de76-33fe-4942-ac44-5ea1d92d8bde\" (UID: \"48e2de76-33fe-4942-ac44-5ea1d92d8bde\") "
Oct 30 19:04:26.487157 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:26.487000    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48e2de76-33fe-4942-ac44-5ea1d92d8bde-kube-api-access-2ktb2\" (UniqueName: \"kubernetes.io/projected/48e2de76-33fe-4942-ac44-5ea1d92d8bde-kube-api-access-2ktb2\") pod \"48e2de76-33fe-4942-ac44-5ea1d92d8bde\" (UID: \"48e2de76-33fe-4942-ac44-5ea1d92d8bde\") "
Oct 30 19:04:26.487369 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:26.487343    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48e2de76-33fe-4942-ac44-5ea1d92d8bde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48e2de76-33fe-4942-ac44-5ea1d92d8bde-maas-api-openshift-ca\") pod \"48e2de76-33fe-4942-ac44-5ea1d92d8bde\" (UID: \"48e2de76-33fe-4942-ac44-5ea1d92d8bde\") "
Oct 30 19:04:26.488472 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:26.488441    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48e2de76-33fe-4942-ac44-5ea1d92d8bde-maas-api-openshift-ca" pod "48e2de76-33fe-4942-ac44-5ea1d92d8bde" (UID: "48e2de76-33fe-4942-ac44-5ea1d92d8bde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:26.489544 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:26.489516    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48e2de76-33fe-4942-ac44-5ea1d92d8bde-maas-api-xks-ca" pod "48e2de76-33fe-4942-ac44-5ea1d92d8bde" (UID: "48e2de76-33fe-4942-ac44-5ea1d92d8bde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:04:26.489827 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:26.489806    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e2de76-33fe-4942-ac44-5ea1d92d8bde-kube-api-access-2ktb2" pod "48e2de76-33fe-4942-ac44-5ea1d92d8bde" (UID: "48e2de76-33fe-4942-ac44-5ea1d92d8bde"). InnerVolumeSpecName "kube-api-access-2ktb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:26.524630 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:26.524594    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:26.589835 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:26.589806    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48e2de76-33fe-4942-ac44-5ea1d92d8bde-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:26.589835 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:26.589834    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48e2de76-33fe-4942-ac44-5ea1d92d8bde-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:26.590008 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:26.589844    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ktb2\" (UniqueName: \"kubernetes.io/projected/48e2de76-33fe-4942-ac44-5ea1d92d8bde-kube-api-access-2ktb2\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:27.200526 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:27.200493    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa834f9-bwffr"
Oct 30 19:04:27.200526 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:27.200518    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa834f9-bwffr" event={"ID":"48e2de76-33fe-4942-ac44-5ea1d92d8bde","Type":"ContainerDied","Data":"1a4e8f025de0fb3efb199dab8f15a2c96aab590368f9937a27515cdf7c8f62ed"}
Oct 30 19:04:27.201017 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:27.200565    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a4e8f025de0fb3efb199dab8f15a2c96aab590368f9937a27515cdf7c8f62ed"
Oct 30 19:04:28.953292 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:28.953249    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:32.069267 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:32.069234    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dbc97cd-7sg7f"]
Oct 30 19:04:32.115254 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:32.115220    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dbc97cd-7sg7f"]
Oct 30 19:04:32.837010 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:32.836976    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="470295aa-9133-4e42-bd40-55d6cb373b03" path="/var/lib/kubelet/pods/470295aa-9133-4e42-bd40-55d6cb373b03/volumes"
Oct 30 19:04:32.966378 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:32.966345    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:33.603459 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:33.603415    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:33.783333 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:33.783285    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:33.953707 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:33.953628    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:34.049027 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:34.048993    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c65470d5-wbdgk"]
Oct 30 19:04:34.095445 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:34.095412    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c65470d5-wbdgk"]
Oct 30 19:04:34.126812 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:34.126779    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:34.852111 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:34.852030    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b428a6a-2a07-4b92-89cf-4646ed10f619" path="/var/lib/kubelet/pods/4b428a6a-2a07-4b92-89cf-4646ed10f619/volumes"
Oct 30 19:04:37.054283 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:37.052166    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-045c23ec-5s4dt"]
Oct 30 19:04:37.098369 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:37.098335    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-045c23ec-5s4dt"]
Oct 30 19:04:37.599998 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:37.599966    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:38.835109 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:38.822519    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:38.883170 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:38.883080    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4afd685c-067a-40ce-becd-80a838afb556" path="/var/lib/kubelet/pods/4afd685c-067a-40ce-becd-80a838afb556/volumes"
Oct 30 19:04:39.480573 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:39.480538    4924 scope.go:122] "RemoveContainer" containerID="ad7003f179408de6a8ff0905bd0ca9ff7444199f027384abd9572077627091a7"
Oct 30 19:04:39.492901 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:39.492875    4924 scope.go:122] "RemoveContainer" containerID="19314710ef7203226e765b9f40974387877c9d2e6198c6eb1d3961b4d750384a"
Oct 30 19:04:39.534111 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:39.534034    4924 scope.go:122] "RemoveContainer" containerID="72069bfc22827e46458e93a452117d1e33c4880f9cb37ea0a08d3da93d5e0f68"
Oct 30 19:04:40.173644 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:40.173616    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:43.661887 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:43.661830    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:43.730265 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:43.730225    4924 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:04:43.730474 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:43.730292    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb" podUID="571509d6-5172-43a0-9bc4-29a0da766ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:04:45.712447 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:04:45.712412    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:04:46.402480 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:46.402450    4924 generic.go:356] "Generic (PLEG): container finished" podID="571509d6-5172-43a0-9bc4-29a0da766ddc" containerID="0388ab91939968b207508aa89dca1f28c77e8bd0354457e0b3fb0a805aa76d89" exitCode=0
Oct 30 19:04:46.402700 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:46.402523    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb" event={"ID":"571509d6-5172-43a0-9bc4-29a0da766ddc","Type":"ContainerDied","Data":"0388ab91939968b207508aa89dca1f28c77e8bd0354457e0b3fb0a805aa76d89"}
Oct 30 19:04:46.403002 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:46.402978    4924 scope.go:122] "RemoveContainer" containerID="0388ab91939968b207508aa89dca1f28c77e8bd0354457e0b3fb0a805aa76d89"
Oct 30 19:04:46.404988 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:46.404955    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:04:46.405099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:46.405049    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:04:46.405166 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:46.405136    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:04:47.446435 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:47.446398    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb" event={"ID":"571509d6-5172-43a0-9bc4-29a0da766ddc","Type":"ContainerStarted","Data":"66cff2ae317acca264fe4144da68fcfed3a2934673a396b9606070f909ff820c"}
Oct 30 19:04:48.264893 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:48.264857    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:04:48.265136 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:48.265113    4924 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 19:04:48.265196 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:48.265179    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb" podUID="571509d6-5172-43a0-9bc4-29a0da766ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 19:04:48.348360 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:48.348322    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:48.522758 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:48.522657    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:48.725790 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:48.725758    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:49.264892 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:49.264859    4924 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 19:04:49.265115 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:49.264924    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb" podUID="571509d6-5172-43a0-9bc4-29a0da766ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 19:04:50.048878 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:50.048841    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49d82f15-8szzf"]
Oct 30 19:04:50.095176 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:50.095147    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49d82f15-8szzf"]
Oct 30 19:04:50.264966 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:50.264934    4924 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 19:04:50.265215 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:50.265002    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb" podUID="571509d6-5172-43a0-9bc4-29a0da766ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 19:04:50.840638 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:50.840596    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a0852d7-de91-4525-b8ad-e4a34abf76a8" path="/var/lib/kubelet/pods/9a0852d7-de91-4525-b8ad-e4a34abf76a8/volumes"
Oct 30 19:04:51.264533 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:51.264496    4924 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 19:04:51.265008 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:51.264556    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb" podUID="571509d6-5172-43a0-9bc4-29a0da766ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 19:04:52.264455 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:52.264416    4924 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 30 19:04:52.264649 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:52.264479    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb" podUID="571509d6-5172-43a0-9bc4-29a0da766ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 30 19:04:53.269087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:53.269036    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:04:53.269496 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:53.269476    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:04:53.270226 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:53.270210    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:04:53.506726 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:53.506688    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:55.121517 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:55.121487    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:55.301025 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:55.300997    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:56.442672 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:56.442636    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:57.063018 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:57.062982    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5a84620-d7lxx"]
Oct 30 19:04:57.107907 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:57.107872    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5a84620-d7lxx"]
Oct 30 19:04:58.285503 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:58.285457    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:58.819854 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:58.819819    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:58.859946 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:58.859914    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c613a857-c27b-4cde-a044-a90005f8f2a3" path="/var/lib/kubelet/pods/c613a857-c27b-4cde-a044-a90005f8f2a3/volumes"
Oct 30 19:04:59.056491 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:59.056457    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-901b4303-2rvkf"]
Oct 30 19:04:59.067171 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:59.067135    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:04:59.100825 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:04:59.100794    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-901b4303-2rvkf"]
Oct 30 19:05:00.837776 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:00.837745    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06839456-517f-4fd9-b469-bb30d10c0e5e" path="/var/lib/kubelet/pods/06839456-517f-4fd9-b469-bb30d10c0e5e/volumes"
Oct 30 19:05:03.281277 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:03.281243    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:03.494324 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:03.494290    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:03.703622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:03.703590    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:05.769768 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:05.769736    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"]
Oct 30 19:05:05.770547 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:05.770530    4924 cpu_manager.go:414] "containerMap: removing container" podUID="48e2de76-33fe-4942-ac44-5ea1d92d8bde" containerName="revoke-keys"
Oct 30 19:05:05.770592 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:05.770549    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e2de76-33fe-4942-ac44-5ea1d92d8bde" containerName="revoke-keys"
Oct 30 19:05:05.770712 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:05.770703    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="48e2de76-33fe-4942-ac44-5ea1d92d8bde" containerName="revoke-keys"
Oct 30 19:05:05.774718 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:05.774699    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:05.880349 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:05.880313    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"]
Oct 30 19:05:05.905929 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:05.905891    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-g5j8l\""
Oct 30 19:05:05.906854 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:05.906820    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2557fb-1287-4606-84d9-266025e6de6a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:05.906854 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:05.906864    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc2557fb-1287-4606-84d9-266025e6de6a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:05.907188 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:05.906989    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2557fb-1287-4606-84d9-266025e6de6a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:05.907188 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:05.907020    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc2557fb-1287-4606-84d9-266025e6de6a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:05.907188 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:05.907141    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f8js\" (UniqueName: \"kubernetes.io/projected/cc2557fb-1287-4606-84d9-266025e6de6a-kube-api-access-8f8js\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:05.907363 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:05.907331    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc2557fb-1287-4606-84d9-266025e6de6a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:05.907424 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:05.907364    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc2557fb-1287-4606-84d9-266025e6de6a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:05.907502 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:05.907484    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc2557fb-1287-4606-84d9-266025e6de6a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:05.907740 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:05.907716    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc2557fb-1287-4606-84d9-266025e6de6a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:06.009327 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.009289    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc2557fb-1287-4606-84d9-266025e6de6a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:06.009540 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.009379    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc2557fb-1287-4606-84d9-266025e6de6a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:06.009540 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.009430    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2557fb-1287-4606-84d9-266025e6de6a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:06.009540 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.009450    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc2557fb-1287-4606-84d9-266025e6de6a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:06.009540 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.009534    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2557fb-1287-4606-84d9-266025e6de6a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:06.009776 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.009565    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc2557fb-1287-4606-84d9-266025e6de6a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:06.009776 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.009655    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f8js\" (UniqueName: \"kubernetes.io/projected/cc2557fb-1287-4606-84d9-266025e6de6a-kube-api-access-8f8js\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:06.009878 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.009827    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2557fb-1287-4606-84d9-266025e6de6a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:06.009878 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.009827    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc2557fb-1287-4606-84d9-266025e6de6a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:06.009995 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.009889    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc2557fb-1287-4606-84d9-266025e6de6a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:06.009995 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.009910    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc2557fb-1287-4606-84d9-266025e6de6a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:06.010218 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.010196    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc2557fb-1287-4606-84d9-266025e6de6a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:06.010360 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.010266    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc2557fb-1287-4606-84d9-266025e6de6a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:06.010360 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.010336    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc2557fb-1287-4606-84d9-266025e6de6a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:06.011845 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.011821    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc2557fb-1287-4606-84d9-266025e6de6a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:06.012401 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.012380    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc2557fb-1287-4606-84d9-266025e6de6a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:06.048732 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.048652    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5215243-8hngd"]
Oct 30 19:05:06.053792 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.053765    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5215243-8hngd"]
Oct 30 19:05:06.063930 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.063907    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc2557fb-1287-4606-84d9-266025e6de6a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:06.064047 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.064000    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f8js\" (UniqueName: \"kubernetes.io/projected/cc2557fb-1287-4606-84d9-266025e6de6a-kube-api-access-8f8js\") pod \"maas-default-gateway-openshift-default-75d7c785f-w2l8h\" (UID: \"cc2557fb-1287-4606-84d9-266025e6de6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:06.089259 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.089230    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:06.388675 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.388647    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:06.536162 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.536123    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"]
Oct 30 19:05:06.569851 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.569807    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:05:06.570135 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.570110    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:05:06.570612 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.570587    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:05:06.585088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.585014    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:06.803484 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.803441    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h" event={"ID":"cc2557fb-1287-4606-84d9-266025e6de6a","Type":"ContainerStarted","Data":"caac0c37193c0d86614f9f0e9b288c841928c861de1be5bbe1514fd897fc39ae"}
Oct 30 19:05:06.803986 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.803519    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h" event={"ID":"cc2557fb-1287-4606-84d9-266025e6de6a","Type":"ContainerStarted","Data":"f93cedfa71bb7fce6fd09f16d8c4881d1e9e855e023f6448b91fc91584499226"}
Oct 30 19:05:06.886675 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:06.886639    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e187191-f813-4a43-95d8-db851971c2ed" path="/var/lib/kubelet/pods/9e187191-f813-4a43-95d8-db851971c2ed/volumes"
Oct 30 19:05:07.090140 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:07.084844    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h" podStartSLOduration=2.084743503 podStartE2EDuration="2.084743503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:07.028294227 +0000 UTC m=+1708.305344672" watchObservedRunningTime="2026-10-30 19:05:07.084743503 +0000 UTC m=+1708.361793948"
Oct 30 19:05:07.107091 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:07.090781    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:07.107091 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:07.098494    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-w2l8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 19:05:07.107091 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:07.098563    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h" podUID="cc2557fb-1287-4606-84d9-266025e6de6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 19:05:08.090437 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:08.090401    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-w2l8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 19:05:08.090925 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:08.090464    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h" podUID="cc2557fb-1287-4606-84d9-266025e6de6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 19:05:08.769110 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:08.767390    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:09.090122 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:09.090028    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-w2l8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 19:05:09.090279 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:09.090119    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h" podUID="cc2557fb-1287-4606-84d9-266025e6de6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 19:05:10.095086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:10.091454    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-w2l8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 19:05:10.095086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:10.091516    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h" podUID="cc2557fb-1287-4606-84d9-266025e6de6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 19:05:11.095529 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:11.095488    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-w2l8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 19:05:11.096242 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:11.096198    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h" podUID="cc2557fb-1287-4606-84d9-266025e6de6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 19:05:12.091347 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:12.090304    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-w2l8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 19:05:12.091347 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:12.090360    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h" podUID="cc2557fb-1287-4606-84d9-266025e6de6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 19:05:12.095198 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:12.095165    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfe70bae-bqfq2"]
Oct 30 19:05:12.100132 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:12.100107    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfe70bae-bqfq2"]
Oct 30 19:05:12.888143 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:12.887999    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27f97819-2d12-424c-b617-936dd6f9099b" path="/var/lib/kubelet/pods/27f97819-2d12-424c-b617-936dd6f9099b/volumes"
Oct 30 19:05:13.089592 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:13.089556    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-w2l8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 19:05:13.089773 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:13.089618    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h" podUID="cc2557fb-1287-4606-84d9-266025e6de6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 19:05:13.860902 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:13.860865    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:14.089856 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:14.089824    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-w2l8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 19:05:14.090029 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:14.089881    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h" podUID="cc2557fb-1287-4606-84d9-266025e6de6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 19:05:15.090207 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:15.090173    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-w2l8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 30 19:05:15.090678 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:15.090643    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h" podUID="cc2557fb-1287-4606-84d9-266025e6de6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 30 19:05:15.747629 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:05:15.747594    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:05:16.772529 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:16.772495    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:16.773439 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:16.773272    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:16.774130 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:16.774104    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w2l8h"
Oct 30 19:05:18.865895 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:18.865803    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:21.660265 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:21.660186    4924 patch_prober.go:29] interesting pod/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:05:21.678554 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:21.660257    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn" podUID="77664ce4-7a08-4b33-bccf-2e0755f13be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:05:22.299302 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:22.299207    4924 generic.go:356] "Generic (PLEG): container finished" podID="91ef253e-1870-4b85-ab78-f155131276a9" containerID="c4d0f33f078f056566b5489b4da4e50807261f8bd73b63603ed899d11fcafdea" exitCode=0
Oct 30 19:05:22.299457 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:22.299405    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4" event={"ID":"91ef253e-1870-4b85-ab78-f155131276a9","Type":"ContainerDied","Data":"c4d0f33f078f056566b5489b4da4e50807261f8bd73b63603ed899d11fcafdea"}
Oct 30 19:05:22.299892 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:22.299865    4924 scope.go:122] "RemoveContainer" containerID="c4d0f33f078f056566b5489b4da4e50807261f8bd73b63603ed899d11fcafdea"
Oct 30 19:05:22.301478 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:22.301456    4924 generic.go:356] "Generic (PLEG): container finished" podID="77664ce4-7a08-4b33-bccf-2e0755f13be3" containerID="fcb1e7fd8d711d7a9fbbaf24aa04a394a144ac4ca60c43d5cadbf94b24d85766" exitCode=0
Oct 30 19:05:22.301586 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:22.301534    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn" event={"ID":"77664ce4-7a08-4b33-bccf-2e0755f13be3","Type":"ContainerDied","Data":"fcb1e7fd8d711d7a9fbbaf24aa04a394a144ac4ca60c43d5cadbf94b24d85766"}
Oct 30 19:05:22.301644 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:22.301616    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:05:22.301716 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:22.301698    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:05:22.301777 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:22.301751    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:05:22.301954 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:22.301938    4924 scope.go:122] "RemoveContainer" containerID="fcb1e7fd8d711d7a9fbbaf24aa04a394a144ac4ca60c43d5cadbf94b24d85766"
Oct 30 19:05:22.303445 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:22.303421    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:05:22.303517 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:22.303488    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:05:22.303555 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:22.303536    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:05:22.449457 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:22.449414    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:22.470447 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:22.470326    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:05:22.864106 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:22.864029    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:23.363336 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:23.363303    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4" event={"ID":"91ef253e-1870-4b85-ab78-f155131276a9","Type":"ContainerStarted","Data":"73cba2b107500a8fcae094cb4ad7fe365780c8def1a937d44d168e50e85dc409"}
Oct 30 19:05:23.366347 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:23.366320    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn" event={"ID":"77664ce4-7a08-4b33-bccf-2e0755f13be3","Type":"ContainerStarted","Data":"87deaaa6e396f6d73ce507587a68176b8ebca2bcf72f35e9ac7291063f496b35"}
Oct 30 19:05:23.708156 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:23.707759    4924 patch_prober.go:29] interesting pod/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:05:23.708156 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:23.707854    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn" podUID="77664ce4-7a08-4b33-bccf-2e0755f13be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:05:23.734115 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:23.714284    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:05:23.782089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:23.769644    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:24.093219 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:24.092211    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:05:24.093718 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:24.093426    4924 patch_prober.go:29] interesting pod/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 19:05:24.093718 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:24.093475    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4" podUID="91ef253e-1870-4b85-ab78-f155131276a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 19:05:24.290626 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:24.290379    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:24.483891 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:24.483838    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:24.691861 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:24.691822    4924 patch_prober.go:29] interesting pod/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:05:24.692105 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:24.691894    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn" podUID="77664ce4-7a08-4b33-bccf-2e0755f13be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:05:24.932708 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:24.917920    4924 patch_prober.go:29] interesting pod/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 19:05:24.932708 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:24.917998    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4" podUID="91ef253e-1870-4b85-ab78-f155131276a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 19:05:25.691389 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:25.691355    4924 patch_prober.go:29] interesting pod/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:05:25.691802 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:25.691409    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn" podUID="77664ce4-7a08-4b33-bccf-2e0755f13be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:05:25.921432 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:25.921384    4924 patch_prober.go:29] interesting pod/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 19:05:25.921715 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:25.921683    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4" podUID="91ef253e-1870-4b85-ab78-f155131276a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 19:05:26.057860 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:26.057779    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1091e678-zj4dk"]
Oct 30 19:05:26.083134 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:26.083096    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1091e678-zj4dk"]
Oct 30 19:05:26.690787 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:26.690749    4924 patch_prober.go:29] interesting pod/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:05:26.690967 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:26.690814    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn" podUID="77664ce4-7a08-4b33-bccf-2e0755f13be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:05:26.776793 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:26.776756    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"]
Oct 30 19:05:26.777634 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:26.777597    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm" podUID="51837408-10f0-4026-b8c6-fcb002a1e459" containerName="istio-proxy" containerID="cri-o://8d0466ae76ba1a576c1de2ecfa36a22c15f4f615c906fcacc789117b67fd2981" gracePeriod=30
Oct 30 19:05:26.894084 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:26.888876    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a843d47b-b14a-4052-8f9b-af32e3db80d7" path="/var/lib/kubelet/pods/a843d47b-b14a-4052-8f9b-af32e3db80d7/volumes"
Oct 30 19:05:26.927207 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:26.927169    4924 patch_prober.go:29] interesting pod/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 19:05:26.965271 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:26.965153    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4" podUID="91ef253e-1870-4b85-ab78-f155131276a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 19:05:27.543089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:27.532629    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:27.690962 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:27.690929    4924 patch_prober.go:29] interesting pod/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:05:27.691156 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:27.690976    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn" podUID="77664ce4-7a08-4b33-bccf-2e0755f13be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:05:27.864596 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:27.864561    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:27.924088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:27.920192    4924 patch_prober.go:29] interesting pod/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 19:05:27.924088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:27.920253    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4" podUID="91ef253e-1870-4b85-ab78-f155131276a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 19:05:28.056430 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:28.056387    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba23f01-627tz"]
Oct 30 19:05:28.090864 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:28.090831    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba23f01-627tz"]
Oct 30 19:05:28.691765 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:28.691728    4924 patch_prober.go:29] interesting pod/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:05:28.691953 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:28.691808    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn" podUID="77664ce4-7a08-4b33-bccf-2e0755f13be3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:05:28.840747 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:28.840708    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9221c05-4591-4b67-b3d3-ccd6bddcff75" path="/var/lib/kubelet/pods/c9221c05-4591-4b67-b3d3-ccd6bddcff75/volumes"
Oct 30 19:05:28.919451 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:28.919422    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:05:28.919944 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:28.919920    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:05:28.920604 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:28.920579    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:05:29.565038 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:29.565002    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:29.695303 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:29.695272    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:05:29.695780 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:29.695762    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:05:29.696462 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:29.696443    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:05:32.072043 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.072008    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:05:32.195396 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.195307    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-workload-certs\") pod \"51837408-10f0-4026-b8c6-fcb002a1e459\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") "
Oct 30 19:05:32.195396 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.195375    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51837408-10f0-4026-b8c6-fcb002a1e459-istio-token\" (UniqueName: \"kubernetes.io/projected/51837408-10f0-4026-b8c6-fcb002a1e459-istio-token\") pod \"51837408-10f0-4026-b8c6-fcb002a1e459\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") "
Oct 30 19:05:32.195609 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.195476    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51837408-10f0-4026-b8c6-fcb002a1e459-kube-api-access-n7gqp\" (UniqueName: \"kubernetes.io/projected/51837408-10f0-4026-b8c6-fcb002a1e459-kube-api-access-n7gqp\") pod \"51837408-10f0-4026-b8c6-fcb002a1e459\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") "
Oct 30 19:05:32.195609 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.195537    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-istio-envoy\") pod \"51837408-10f0-4026-b8c6-fcb002a1e459\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") "
Oct 30 19:05:32.195728 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.195617    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-workload-certs" pod "51837408-10f0-4026-b8c6-fcb002a1e459" (UID: "51837408-10f0-4026-b8c6-fcb002a1e459"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:32.195728 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.195646    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-credential-socket\") pod \"51837408-10f0-4026-b8c6-fcb002a1e459\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") "
Oct 30 19:05:32.195728 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.195703    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-workload-socket\") pod \"51837408-10f0-4026-b8c6-fcb002a1e459\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") "
Oct 30 19:05:32.196100 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.195898    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51837408-10f0-4026-b8c6-fcb002a1e459-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51837408-10f0-4026-b8c6-fcb002a1e459-istiod-ca-cert\") pod \"51837408-10f0-4026-b8c6-fcb002a1e459\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") "
Oct 30 19:05:32.196100 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.195929    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-credential-socket" pod "51837408-10f0-4026-b8c6-fcb002a1e459" (UID: "51837408-10f0-4026-b8c6-fcb002a1e459"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:32.196100 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.196009    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51837408-10f0-4026-b8c6-fcb002a1e459-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51837408-10f0-4026-b8c6-fcb002a1e459-istio-podinfo\") pod \"51837408-10f0-4026-b8c6-fcb002a1e459\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") "
Oct 30 19:05:32.196320 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.196105    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-istio-data\") pod \"51837408-10f0-4026-b8c6-fcb002a1e459\" (UID: \"51837408-10f0-4026-b8c6-fcb002a1e459\") "
Oct 30 19:05:32.196371 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.196336    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51837408-10f0-4026-b8c6-fcb002a1e459-istiod-ca-cert" pod "51837408-10f0-4026-b8c6-fcb002a1e459" (UID: "51837408-10f0-4026-b8c6-fcb002a1e459"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:32.196596 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.196466    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-workload-socket" pod "51837408-10f0-4026-b8c6-fcb002a1e459" (UID: "51837408-10f0-4026-b8c6-fcb002a1e459"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:32.197116 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.196682    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-istio-data" pod "51837408-10f0-4026-b8c6-fcb002a1e459" (UID: "51837408-10f0-4026-b8c6-fcb002a1e459"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:32.197116 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.197027    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:32.197116 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.197045    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:32.197116 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.197054    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51837408-10f0-4026-b8c6-fcb002a1e459-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:32.197116 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.197082    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:32.197116 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.197095    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:32.199114 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.199057    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51837408-10f0-4026-b8c6-fcb002a1e459-istio-token" pod "51837408-10f0-4026-b8c6-fcb002a1e459" (UID: "51837408-10f0-4026-b8c6-fcb002a1e459"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:32.199308 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.199290    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51837408-10f0-4026-b8c6-fcb002a1e459-istio-podinfo" pod "51837408-10f0-4026-b8c6-fcb002a1e459" (UID: "51837408-10f0-4026-b8c6-fcb002a1e459"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:05:32.199642 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.199616    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-istio-envoy" pod "51837408-10f0-4026-b8c6-fcb002a1e459" (UID: "51837408-10f0-4026-b8c6-fcb002a1e459"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:32.199802 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.199780    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51837408-10f0-4026-b8c6-fcb002a1e459-kube-api-access-n7gqp" pod "51837408-10f0-4026-b8c6-fcb002a1e459" (UID: "51837408-10f0-4026-b8c6-fcb002a1e459"). InnerVolumeSpecName "kube-api-access-n7gqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:32.283016 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.282984    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:32.299012 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.298984    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51837408-10f0-4026-b8c6-fcb002a1e459-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:32.299012 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.299011    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51837408-10f0-4026-b8c6-fcb002a1e459-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:32.299195 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.299021    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7gqp\" (UniqueName: \"kubernetes.io/projected/51837408-10f0-4026-b8c6-fcb002a1e459-kube-api-access-n7gqp\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:32.299195 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.299032    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51837408-10f0-4026-b8c6-fcb002a1e459-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:32.485151 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.484571    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:32.531984 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.531952    4924 generic.go:356] "Generic (PLEG): container finished" podID="51837408-10f0-4026-b8c6-fcb002a1e459" containerID="8d0466ae76ba1a576c1de2ecfa36a22c15f4f615c906fcacc789117b67fd2981" exitCode=0
Oct 30 19:05:32.532240 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.532002    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"
Oct 30 19:05:32.532240 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.532030    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm" event={"ID":"51837408-10f0-4026-b8c6-fcb002a1e459","Type":"ContainerDied","Data":"8d0466ae76ba1a576c1de2ecfa36a22c15f4f615c906fcacc789117b67fd2981"}
Oct 30 19:05:32.532240 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.532130    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm" event={"ID":"51837408-10f0-4026-b8c6-fcb002a1e459","Type":"ContainerDied","Data":"0c0d5d3e741c9e90dcac8f6a35b90ce080adac9e95abd83df7c6749b36d2c263"}
Oct 30 19:05:32.532240 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.532161    4924 scope.go:122] "RemoveContainer" containerID="8d0466ae76ba1a576c1de2ecfa36a22c15f4f615c906fcacc789117b67fd2981"
Oct 30 19:05:32.565197 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.565176    4924 scope.go:122] "RemoveContainer" containerID="8d0466ae76ba1a576c1de2ecfa36a22c15f4f615c906fcacc789117b67fd2981"
Oct 30 19:05:32.565524 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:05:32.565503    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d0466ae76ba1a576c1de2ecfa36a22c15f4f615c906fcacc789117b67fd2981\": container with ID starting with 8d0466ae76ba1a576c1de2ecfa36a22c15f4f615c906fcacc789117b67fd2981 not found: ID does not exist" containerID="8d0466ae76ba1a576c1de2ecfa36a22c15f4f615c906fcacc789117b67fd2981"
Oct 30 19:05:32.565580 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.565532    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d0466ae76ba1a576c1de2ecfa36a22c15f4f615c906fcacc789117b67fd2981"} err="failed to get container status \"8d0466ae76ba1a576c1de2ecfa36a22c15f4f615c906fcacc789117b67fd2981\": rpc error: code = NotFound desc = could not find container \"8d0466ae76ba1a576c1de2ecfa36a22c15f4f615c906fcacc789117b67fd2981\": container with ID starting with 8d0466ae76ba1a576c1de2ecfa36a22c15f4f615c906fcacc789117b67fd2981 not found: ID does not exist"
Oct 30 19:05:32.712407 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.712368    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"]
Oct 30 19:05:32.752138 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.751619    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6r4dm"]
Oct 30 19:05:32.859087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:32.851778    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51837408-10f0-4026-b8c6-fcb002a1e459" path="/var/lib/kubelet/pods/51837408-10f0-4026-b8c6-fcb002a1e459/volumes"
Oct 30 19:05:34.865840 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:34.865778    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:37.050880 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:37.050848    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4332bcf9-j2qlr"]
Oct 30 19:05:37.097115 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:37.097081    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4332bcf9-j2qlr"]
Oct 30 19:05:37.944038 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:37.944010    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:38.050945 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:38.050910    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b97aee-fknpv"]
Oct 30 19:05:38.096586 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:38.096555    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b97aee-fknpv"]
Oct 30 19:05:38.838989 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:38.838957    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b34001f-f608-427e-ae9d-b304dcbea464" path="/var/lib/kubelet/pods/5b34001f-f608-427e-ae9d-b304dcbea464/volumes"
Oct 30 19:05:38.839552 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:38.839535    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3ef59b7-2567-49bb-ae3a-633348290767" path="/var/lib/kubelet/pods/f3ef59b7-2567-49bb-ae3a-633348290767/volumes"
Oct 30 19:05:39.633960 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:39.633926    4924 scope.go:122] "RemoveContainer" containerID="187eab82d11a8011329b5c35f49eac97fea68789091622aada4b101bb17ea909"
Oct 30 19:05:39.645210 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:39.645192    4924 scope.go:122] "RemoveContainer" containerID="e17ab1ea861343ad922ed51fe6ede69e09a3a9a08f0e88ccbdeb9c032ac544e5"
Oct 30 19:05:39.655534 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:39.655516    4924 scope.go:122] "RemoveContainer" containerID="dfb41da98e66c3f09f405eaea91c44e26b048e525b130000f9c0bd10ff882df2"
Oct 30 19:05:39.665340 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:39.665319    4924 scope.go:122] "RemoveContainer" containerID="2f69f6399b814ebdd088052e7553c28f48565f9463081f4d74c89ea1e08e1183"
Oct 30 19:05:39.675085 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:39.675055    4924 scope.go:122] "RemoveContainer" containerID="0c9cbbbcbeb401cac5e2f7b6d6d9aa3457344e6c6cd5f282f5e762fc27e0ae5d"
Oct 30 19:05:39.684095 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:39.684044    4924 scope.go:122] "RemoveContainer" containerID="6a15926b7c2ffd225c8207060f38dca2345439901a444c12c26ba0fd0e05c344"
Oct 30 19:05:39.696385 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:39.694246    4924 scope.go:122] "RemoveContainer" containerID="632b1c35d54836a1ec02f5c024a4f6587eb23702d04a4a56c2ce2b792e60853f"
Oct 30 19:05:39.705994 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:39.705979    4924 scope.go:122] "RemoveContainer" containerID="9a208834f15043265a1273ff093d0e932caa2f881ec11cfdf89225207f4c0959"
Oct 30 19:05:39.936643 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:39.936542    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:42.476805 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:42.476770    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:42.746754 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:42.746678    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:42.997548 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:42.997466    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:44.143747 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:44.143714    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:44.539085 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:44.538955    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:44.924634 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:44.924605    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:45.140982 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:45.140936    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:45.652300 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:45.652217    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:45.715179 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:05:45.715143    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:05:46.315606 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:46.315558    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8416e0-r2sv6"]
Oct 30 19:05:46.316379 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:46.316360    4924 cpu_manager.go:414] "containerMap: removing container" podUID="51837408-10f0-4026-b8c6-fcb002a1e459" containerName="istio-proxy"
Oct 30 19:05:46.316379 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:46.316378    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51837408-10f0-4026-b8c6-fcb002a1e459" containerName="istio-proxy"
Oct 30 19:05:46.316524 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:46.316514    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="51837408-10f0-4026-b8c6-fcb002a1e459" containerName="istio-proxy"
Oct 30 19:05:46.320786 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:46.320770    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8416e0-r2sv6"
Oct 30 19:05:46.378145 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:46.378118    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8416e0-r2sv6"]
Oct 30 19:05:46.418118 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:46.418051    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:05:46.418300 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:46.418122    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:05:46.418300 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:46.418130    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:05:46.418456 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:46.418347    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:05:46.454573 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:46.454542    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n65kc\" (UniqueName: \"kubernetes.io/projected/a9cce67a-d961-4ba6-92d9-df9b2bea4919-kube-api-access-n65kc\") pod \"maas-api-revoke-sub-ed8416e0-r2sv6\" (UID: \"a9cce67a-d961-4ba6-92d9-df9b2bea4919\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8416e0-r2sv6"
Oct 30 19:05:46.454810 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:46.454790    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9cce67a-d961-4ba6-92d9-df9b2bea4919-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed8416e0-r2sv6\" (UID: \"a9cce67a-d961-4ba6-92d9-df9b2bea4919\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8416e0-r2sv6"
Oct 30 19:05:46.454897 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:46.454826    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9cce67a-d961-4ba6-92d9-df9b2bea4919-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed8416e0-r2sv6\" (UID: \"a9cce67a-d961-4ba6-92d9-df9b2bea4919\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8416e0-r2sv6"
Oct 30 19:05:46.555951 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:46.555917    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9cce67a-d961-4ba6-92d9-df9b2bea4919-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed8416e0-r2sv6\" (UID: \"a9cce67a-d961-4ba6-92d9-df9b2bea4919\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8416e0-r2sv6"
Oct 30 19:05:46.556147 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:46.555961    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9cce67a-d961-4ba6-92d9-df9b2bea4919-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed8416e0-r2sv6\" (UID: \"a9cce67a-d961-4ba6-92d9-df9b2bea4919\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8416e0-r2sv6"
Oct 30 19:05:46.556147 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:46.556130    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n65kc\" (UniqueName: \"kubernetes.io/projected/a9cce67a-d961-4ba6-92d9-df9b2bea4919-kube-api-access-n65kc\") pod \"maas-api-revoke-sub-ed8416e0-r2sv6\" (UID: \"a9cce67a-d961-4ba6-92d9-df9b2bea4919\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8416e0-r2sv6"
Oct 30 19:05:46.556773 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:46.556744    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9cce67a-d961-4ba6-92d9-df9b2bea4919-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed8416e0-r2sv6\" (UID: \"a9cce67a-d961-4ba6-92d9-df9b2bea4919\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8416e0-r2sv6"
Oct 30 19:05:46.558464 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:46.558438    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9cce67a-d961-4ba6-92d9-df9b2bea4919-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed8416e0-r2sv6\" (UID: \"a9cce67a-d961-4ba6-92d9-df9b2bea4919\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8416e0-r2sv6"
Oct 30 19:05:46.614360 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:46.614334    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n65kc\" (UniqueName: \"kubernetes.io/projected/a9cce67a-d961-4ba6-92d9-df9b2bea4919-kube-api-access-n65kc\") pod \"maas-api-revoke-sub-ed8416e0-r2sv6\" (UID: \"a9cce67a-d961-4ba6-92d9-df9b2bea4919\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8416e0-r2sv6"
Oct 30 19:05:46.633399 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:46.633374    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8416e0-r2sv6"
Oct 30 19:05:47.015114 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:05:47.015083    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9cce67a_d961_4ba6_92d9_df9b2bea4919.slice/crio-cc552eda36480d229f39386ed76423815633155ad981815f154050445e827352 WatchSource:0}: Error finding container cc552eda36480d229f39386ed76423815633155ad981815f154050445e827352: Status 404 returned error can't find the container with id cc552eda36480d229f39386ed76423815633155ad981815f154050445e827352
Oct 30 19:05:47.015702 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:47.015673    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8416e0-r2sv6"]
Oct 30 19:05:47.673521 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:47.673432    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8416e0-r2sv6" event={"ID":"a9cce67a-d961-4ba6-92d9-df9b2bea4919","Type":"ContainerStarted","Data":"979b597cee473b38b283bdf82749226bc4955e3a6bbe5617f24f8cd553c62e97"}
Oct 30 19:05:47.673521 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:47.673492    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8416e0-r2sv6" event={"ID":"a9cce67a-d961-4ba6-92d9-df9b2bea4919","Type":"ContainerStarted","Data":"cc552eda36480d229f39386ed76423815633155ad981815f154050445e827352"}
Oct 30 19:05:47.785138 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:47.785082    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8416e0-r2sv6" podStartSLOduration=1.7850569790000002 podStartE2EDuration="1.785056979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:47.782282218 +0000 UTC m=+1749.059332663" watchObservedRunningTime="2026-10-30 19:05:47.785056979 +0000 UTC m=+1749.062107424"
Oct 30 19:05:48.683790 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:48.683756    4924 generic.go:356] "Generic (PLEG): container finished" podID="a9cce67a-d961-4ba6-92d9-df9b2bea4919" containerID="979b597cee473b38b283bdf82749226bc4955e3a6bbe5617f24f8cd553c62e97" exitCode=0
Oct 30 19:05:48.684187 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:48.683841    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8416e0-r2sv6" event={"ID":"a9cce67a-d961-4ba6-92d9-df9b2bea4919","Type":"ContainerDied","Data":"979b597cee473b38b283bdf82749226bc4955e3a6bbe5617f24f8cd553c62e97"}
Oct 30 19:05:48.879403 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:48.879372    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:49.173624 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:49.173565    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:49.871621 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:49.871597    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8416e0-r2sv6"
Oct 30 19:05:50.014670 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:50.014575    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9cce67a-d961-4ba6-92d9-df9b2bea4919-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9cce67a-d961-4ba6-92d9-df9b2bea4919-maas-api-xks-ca\") pod \"a9cce67a-d961-4ba6-92d9-df9b2bea4919\" (UID: \"a9cce67a-d961-4ba6-92d9-df9b2bea4919\") "
Oct 30 19:05:50.014846 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:50.014719    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9cce67a-d961-4ba6-92d9-df9b2bea4919-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9cce67a-d961-4ba6-92d9-df9b2bea4919-maas-api-openshift-ca\") pod \"a9cce67a-d961-4ba6-92d9-df9b2bea4919\" (UID: \"a9cce67a-d961-4ba6-92d9-df9b2bea4919\") "
Oct 30 19:05:50.014846 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:50.014800    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9cce67a-d961-4ba6-92d9-df9b2bea4919-kube-api-access-n65kc\" (UniqueName: \"kubernetes.io/projected/a9cce67a-d961-4ba6-92d9-df9b2bea4919-kube-api-access-n65kc\") pod \"a9cce67a-d961-4ba6-92d9-df9b2bea4919\" (UID: \"a9cce67a-d961-4ba6-92d9-df9b2bea4919\") "
Oct 30 19:05:50.015229 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:50.015182    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9cce67a-d961-4ba6-92d9-df9b2bea4919-maas-api-openshift-ca" pod "a9cce67a-d961-4ba6-92d9-df9b2bea4919" (UID: "a9cce67a-d961-4ba6-92d9-df9b2bea4919"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:50.015665 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:50.015640    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9cce67a-d961-4ba6-92d9-df9b2bea4919-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:50.017344 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:50.017316    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9cce67a-d961-4ba6-92d9-df9b2bea4919-maas-api-xks-ca" pod "a9cce67a-d961-4ba6-92d9-df9b2bea4919" (UID: "a9cce67a-d961-4ba6-92d9-df9b2bea4919"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:50.017612 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:50.017591    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9cce67a-d961-4ba6-92d9-df9b2bea4919-kube-api-access-n65kc" pod "a9cce67a-d961-4ba6-92d9-df9b2bea4919" (UID: "a9cce67a-d961-4ba6-92d9-df9b2bea4919"). InnerVolumeSpecName "kube-api-access-n65kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:50.116570 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:50.116540    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9cce67a-d961-4ba6-92d9-df9b2bea4919-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:50.116570 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:50.116568    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n65kc\" (UniqueName: \"kubernetes.io/projected/a9cce67a-d961-4ba6-92d9-df9b2bea4919-kube-api-access-n65kc\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:50.701195 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:50.701160    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8416e0-r2sv6" event={"ID":"a9cce67a-d961-4ba6-92d9-df9b2bea4919","Type":"ContainerDied","Data":"cc552eda36480d229f39386ed76423815633155ad981815f154050445e827352"}
Oct 30 19:05:50.701195 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:50.701181    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed8416e0-r2sv6"
Oct 30 19:05:50.701376 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:50.701208    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc552eda36480d229f39386ed76423815633155ad981815f154050445e827352"
Oct 30 19:05:50.970490 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:50.970407    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:51.970495 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:51.970452    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:52.130785 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:52.130745    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f332265-2kkl4"]
Oct 30 19:05:52.135903 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:52.135869    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f332265-2kkl4"]
Oct 30 19:05:52.205036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:52.205000    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:52.839413 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:52.839379    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca5ef146-ff29-46ed-b391-6c1962a4c6bf" path="/var/lib/kubelet/pods/ca5ef146-ff29-46ed-b391-6c1962a4c6bf/volumes"
Oct 30 19:05:53.051500 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:53.051466    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea593430-2gt89"]
Oct 30 19:05:53.097945 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:53.097922    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea593430-2gt89"]
Oct 30 19:05:53.109602 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:53.109580    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cef683f-fvknj"]
Oct 30 19:05:53.110392 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:53.110374    4924 cpu_manager.go:414] "containerMap: removing container" podUID="a9cce67a-d961-4ba6-92d9-df9b2bea4919" containerName="revoke-keys"
Oct 30 19:05:53.110474 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:53.110394    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9cce67a-d961-4ba6-92d9-df9b2bea4919" containerName="revoke-keys"
Oct 30 19:05:53.110548 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:53.110538    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9cce67a-d961-4ba6-92d9-df9b2bea4919" containerName="revoke-keys"
Oct 30 19:05:53.114090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:53.114059    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cef683f-fvknj"
Oct 30 19:05:53.164273 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:53.164104    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:05:53.164273 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:53.164107    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:05:53.164273 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:53.164149    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:05:53.164273 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:53.164172    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:05:53.170276 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:53.170249    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cef683f-fvknj"]
Oct 30 19:05:53.286897 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:53.286858    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl9mw\" (UniqueName: \"kubernetes.io/projected/2cb370d9-c034-45eb-862c-20d8fe198847-kube-api-access-zl9mw\") pod \"maas-api-revoke-keys-e2e-derive-4cef683f-fvknj\" (UID: \"2cb370d9-c034-45eb-862c-20d8fe198847\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cef683f-fvknj"
Oct 30 19:05:53.287112 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:53.287052    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb370d9-c034-45eb-862c-20d8fe198847-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4cef683f-fvknj\" (UID: \"2cb370d9-c034-45eb-862c-20d8fe198847\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cef683f-fvknj"
Oct 30 19:05:53.287180 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:53.287155    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb370d9-c034-45eb-862c-20d8fe198847-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4cef683f-fvknj\" (UID: \"2cb370d9-c034-45eb-862c-20d8fe198847\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cef683f-fvknj"
Oct 30 19:05:53.389310 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:53.389228    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl9mw\" (UniqueName: \"kubernetes.io/projected/2cb370d9-c034-45eb-862c-20d8fe198847-kube-api-access-zl9mw\") pod \"maas-api-revoke-keys-e2e-derive-4cef683f-fvknj\" (UID: \"2cb370d9-c034-45eb-862c-20d8fe198847\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cef683f-fvknj"
Oct 30 19:05:53.389475 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:53.389359    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb370d9-c034-45eb-862c-20d8fe198847-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4cef683f-fvknj\" (UID: \"2cb370d9-c034-45eb-862c-20d8fe198847\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cef683f-fvknj"
Oct 30 19:05:53.389475 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:53.389418    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb370d9-c034-45eb-862c-20d8fe198847-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4cef683f-fvknj\" (UID: \"2cb370d9-c034-45eb-862c-20d8fe198847\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cef683f-fvknj"
Oct 30 19:05:53.389988 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:53.389965    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb370d9-c034-45eb-862c-20d8fe198847-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4cef683f-fvknj\" (UID: \"2cb370d9-c034-45eb-862c-20d8fe198847\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cef683f-fvknj"
Oct 30 19:05:53.391757 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:53.391736    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb370d9-c034-45eb-862c-20d8fe198847-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4cef683f-fvknj\" (UID: \"2cb370d9-c034-45eb-862c-20d8fe198847\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cef683f-fvknj"
Oct 30 19:05:53.444261 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:53.444237    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl9mw\" (UniqueName: \"kubernetes.io/projected/2cb370d9-c034-45eb-862c-20d8fe198847-kube-api-access-zl9mw\") pod \"maas-api-revoke-keys-e2e-derive-4cef683f-fvknj\" (UID: \"2cb370d9-c034-45eb-862c-20d8fe198847\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cef683f-fvknj"
Oct 30 19:05:53.728723 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:53.728639    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cef683f-fvknj"
Oct 30 19:05:54.113668 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:54.113635    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cef683f-fvknj"]
Oct 30 19:05:54.116140 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:05:54.116053    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb370d9_c034_45eb_862c_20d8fe198847.slice/crio-c1488478baecc5ba180e588296bfdc6aa9da3ac374f39a3e18377442e318a3d9 WatchSource:0}: Error finding container c1488478baecc5ba180e588296bfdc6aa9da3ac374f39a3e18377442e318a3d9: Status 404 returned error can't find the container with id c1488478baecc5ba180e588296bfdc6aa9da3ac374f39a3e18377442e318a3d9
Oct 30 19:05:54.749058 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:54.748949    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cef683f-fvknj" event={"ID":"2cb370d9-c034-45eb-862c-20d8fe198847","Type":"ContainerStarted","Data":"38ecb211ab7e6dae475bc9a3124ffbcb3f352de6d1fab2fae4fcc252e32522de"}
Oct 30 19:05:54.749277 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:54.749098    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cef683f-fvknj" event={"ID":"2cb370d9-c034-45eb-862c-20d8fe198847","Type":"ContainerStarted","Data":"c1488478baecc5ba180e588296bfdc6aa9da3ac374f39a3e18377442e318a3d9"}
Oct 30 19:05:54.837452 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:54.837413    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="873be518-74f7-44f7-b547-5626b97ebfad" path="/var/lib/kubelet/pods/873be518-74f7-44f7-b547-5626b97ebfad/volumes"
Oct 30 19:05:54.861952 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:54.861904    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cef683f-fvknj" podStartSLOduration=1.861890776 podStartE2EDuration="1.861890776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:54.857749128 +0000 UTC m=+1756.134799583" watchObservedRunningTime="2026-10-30 19:05:54.861890776 +0000 UTC m=+1756.138941237"
Oct 30 19:05:55.773089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:55.769053    4924 generic.go:356] "Generic (PLEG): container finished" podID="2cb370d9-c034-45eb-862c-20d8fe198847" containerID="38ecb211ab7e6dae475bc9a3124ffbcb3f352de6d1fab2fae4fcc252e32522de" exitCode=0
Oct 30 19:05:55.773089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:55.769121    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cef683f-fvknj" event={"ID":"2cb370d9-c034-45eb-862c-20d8fe198847","Type":"ContainerDied","Data":"38ecb211ab7e6dae475bc9a3124ffbcb3f352de6d1fab2fae4fcc252e32522de"}
Oct 30 19:05:56.473573 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:56.473537    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:57.016692 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.016665    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cef683f-fvknj"
Oct 30 19:05:57.071698 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.071619    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cb370d9-c034-45eb-862c-20d8fe198847-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb370d9-c034-45eb-862c-20d8fe198847-maas-api-xks-ca\") pod \"2cb370d9-c034-45eb-862c-20d8fe198847\" (UID: \"2cb370d9-c034-45eb-862c-20d8fe198847\") "
Oct 30 19:05:57.071698 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.071680    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb370d9-c034-45eb-862c-20d8fe198847-kube-api-access-zl9mw\" (UniqueName: \"kubernetes.io/projected/2cb370d9-c034-45eb-862c-20d8fe198847-kube-api-access-zl9mw\") pod \"2cb370d9-c034-45eb-862c-20d8fe198847\" (UID: \"2cb370d9-c034-45eb-862c-20d8fe198847\") "
Oct 30 19:05:57.071928 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.071773    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cb370d9-c034-45eb-862c-20d8fe198847-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb370d9-c034-45eb-862c-20d8fe198847-maas-api-openshift-ca\") pod \"2cb370d9-c034-45eb-862c-20d8fe198847\" (UID: \"2cb370d9-c034-45eb-862c-20d8fe198847\") "
Oct 30 19:05:57.072117 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.072090    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb370d9-c034-45eb-862c-20d8fe198847-maas-api-openshift-ca" pod "2cb370d9-c034-45eb-862c-20d8fe198847" (UID: "2cb370d9-c034-45eb-862c-20d8fe198847"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:57.072792 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.072774    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb370d9-c034-45eb-862c-20d8fe198847-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:57.073904 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.073876    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cb370d9-c034-45eb-862c-20d8fe198847-maas-api-xks-ca" pod "2cb370d9-c034-45eb-862c-20d8fe198847" (UID: "2cb370d9-c034-45eb-862c-20d8fe198847"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:57.073904 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.073898    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb370d9-c034-45eb-862c-20d8fe198847-kube-api-access-zl9mw" pod "2cb370d9-c034-45eb-862c-20d8fe198847" (UID: "2cb370d9-c034-45eb-862c-20d8fe198847"). InnerVolumeSpecName "kube-api-access-zl9mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:57.101627 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.101602    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:05:57.175254 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.175216    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb370d9-c034-45eb-862c-20d8fe198847-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:57.175254 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.175251    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl9mw\" (UniqueName: \"kubernetes.io/projected/2cb370d9-c034-45eb-862c-20d8fe198847-kube-api-access-zl9mw\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:57.180274 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.180242    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8"]
Oct 30 19:05:57.181456 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.181429    4924 cpu_manager.go:414] "containerMap: removing container" podUID="2cb370d9-c034-45eb-862c-20d8fe198847" containerName="revoke-keys"
Oct 30 19:05:57.181456 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.181453    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb370d9-c034-45eb-862c-20d8fe198847" containerName="revoke-keys"
Oct 30 19:05:57.181660 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.181621    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb370d9-c034-45eb-862c-20d8fe198847" containerName="revoke-keys"
Oct 30 19:05:57.186449 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.184974    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8"
Oct 30 19:05:57.254414 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.254127    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8"]
Oct 30 19:05:57.277090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.277040    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38924c59-63d4-40d5-9a35-204dcd3d4550-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8\" (UID: \"38924c59-63d4-40d5-9a35-204dcd3d4550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8"
Oct 30 19:05:57.277238 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.277201    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxxf4\" (UniqueName: \"kubernetes.io/projected/38924c59-63d4-40d5-9a35-204dcd3d4550-kube-api-access-qxxf4\") pod \"maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8\" (UID: \"38924c59-63d4-40d5-9a35-204dcd3d4550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8"
Oct 30 19:05:57.277413 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.277396    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38924c59-63d4-40d5-9a35-204dcd3d4550-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8\" (UID: \"38924c59-63d4-40d5-9a35-204dcd3d4550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8"
Oct 30 19:05:57.378655 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.378623    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38924c59-63d4-40d5-9a35-204dcd3d4550-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8\" (UID: \"38924c59-63d4-40d5-9a35-204dcd3d4550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8"
Oct 30 19:05:57.378849 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.378756    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38924c59-63d4-40d5-9a35-204dcd3d4550-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8\" (UID: \"38924c59-63d4-40d5-9a35-204dcd3d4550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8"
Oct 30 19:05:57.378849 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.378801    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxxf4\" (UniqueName: \"kubernetes.io/projected/38924c59-63d4-40d5-9a35-204dcd3d4550-kube-api-access-qxxf4\") pod \"maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8\" (UID: \"38924c59-63d4-40d5-9a35-204dcd3d4550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8"
Oct 30 19:05:57.379478 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.379455    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38924c59-63d4-40d5-9a35-204dcd3d4550-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8\" (UID: \"38924c59-63d4-40d5-9a35-204dcd3d4550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8"
Oct 30 19:05:57.381163 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.381139    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38924c59-63d4-40d5-9a35-204dcd3d4550-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8\" (UID: \"38924c59-63d4-40d5-9a35-204dcd3d4550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8"
Oct 30 19:05:57.433697 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.433673    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxxf4\" (UniqueName: \"kubernetes.io/projected/38924c59-63d4-40d5-9a35-204dcd3d4550-kube-api-access-qxxf4\") pod \"maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8\" (UID: \"38924c59-63d4-40d5-9a35-204dcd3d4550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8"
Oct 30 19:05:57.502527 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.502481    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8"
Oct 30 19:05:57.787628 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.787541    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cef683f-fvknj" event={"ID":"2cb370d9-c034-45eb-862c-20d8fe198847","Type":"ContainerDied","Data":"c1488478baecc5ba180e588296bfdc6aa9da3ac374f39a3e18377442e318a3d9"}
Oct 30 19:05:57.787628 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.787582    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cef683f-fvknj"
Oct 30 19:05:57.787628 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.787617    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1488478baecc5ba180e588296bfdc6aa9da3ac374f39a3e18377442e318a3d9"
Oct 30 19:05:57.879507 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:05:57.879471    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38924c59_63d4_40d5_9a35_204dcd3d4550.slice/crio-c85e6656fbe0a4e399f58ea14060254ffa8765234c30f79f4e8c5d7f41aa8cc1 WatchSource:0}: Error finding container c85e6656fbe0a4e399f58ea14060254ffa8765234c30f79f4e8c5d7f41aa8cc1: Status 404 returned error can't find the container with id c85e6656fbe0a4e399f58ea14060254ffa8765234c30f79f4e8c5d7f41aa8cc1
Oct 30 19:05:57.879868 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:57.879844    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8"]
Oct 30 19:05:58.075854 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:58.075768    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9799318d-zrb9d"]
Oct 30 19:05:58.080588 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:58.080560    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9799318d-zrb9d"]
Oct 30 19:05:58.802020 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:58.801984    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8" event={"ID":"38924c59-63d4-40d5-9a35-204dcd3d4550","Type":"ContainerStarted","Data":"6bc5455ea3e5342c66ecfce09c07f30d5c3e0d82fdf560f54272bafce8033a39"}
Oct 30 19:05:58.802253 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:58.802052    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8" event={"ID":"38924c59-63d4-40d5-9a35-204dcd3d4550","Type":"ContainerStarted","Data":"c85e6656fbe0a4e399f58ea14060254ffa8765234c30f79f4e8c5d7f41aa8cc1"}
Oct 30 19:05:58.837482 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:58.837452    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fe3ad08-26ac-4e3d-868b-4a4e69706985" path="/var/lib/kubelet/pods/0fe3ad08-26ac-4e3d-868b-4a4e69706985/volumes"
Oct 30 19:05:58.933123 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:58.933050    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8" podStartSLOduration=1.9330368500000001 podStartE2EDuration="1.93303685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:58.929140535 +0000 UTC m=+1760.206190981" watchObservedRunningTime="2026-10-30 19:05:58.93303685 +0000 UTC m=+1760.210087319"
Oct 30 19:05:59.814721 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:59.814682    4924 generic.go:356] "Generic (PLEG): container finished" podID="38924c59-63d4-40d5-9a35-204dcd3d4550" containerID="6bc5455ea3e5342c66ecfce09c07f30d5c3e0d82fdf560f54272bafce8033a39" exitCode=0
Oct 30 19:05:59.815278 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:05:59.814742    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8" event={"ID":"38924c59-63d4-40d5-9a35-204dcd3d4550","Type":"ContainerDied","Data":"6bc5455ea3e5342c66ecfce09c07f30d5c3e0d82fdf560f54272bafce8033a39"}
Oct 30 19:06:01.008960 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:01.008935    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8"
Oct 30 19:06:01.038282 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:01.038260    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38924c59-63d4-40d5-9a35-204dcd3d4550-kube-api-access-qxxf4\" (UniqueName: \"kubernetes.io/projected/38924c59-63d4-40d5-9a35-204dcd3d4550-kube-api-access-qxxf4\") pod \"38924c59-63d4-40d5-9a35-204dcd3d4550\" (UID: \"38924c59-63d4-40d5-9a35-204dcd3d4550\") "
Oct 30 19:06:01.038399 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:01.038301    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38924c59-63d4-40d5-9a35-204dcd3d4550-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38924c59-63d4-40d5-9a35-204dcd3d4550-maas-api-openshift-ca\") pod \"38924c59-63d4-40d5-9a35-204dcd3d4550\" (UID: \"38924c59-63d4-40d5-9a35-204dcd3d4550\") "
Oct 30 19:06:01.038577 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:01.038556    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38924c59-63d4-40d5-9a35-204dcd3d4550-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38924c59-63d4-40d5-9a35-204dcd3d4550-maas-api-xks-ca\") pod \"38924c59-63d4-40d5-9a35-204dcd3d4550\" (UID: \"38924c59-63d4-40d5-9a35-204dcd3d4550\") "
Oct 30 19:06:01.038739 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:01.038718    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38924c59-63d4-40d5-9a35-204dcd3d4550-maas-api-openshift-ca" pod "38924c59-63d4-40d5-9a35-204dcd3d4550" (UID: "38924c59-63d4-40d5-9a35-204dcd3d4550"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:01.039195 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:01.039175    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38924c59-63d4-40d5-9a35-204dcd3d4550-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:01.040408 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:01.040376    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38924c59-63d4-40d5-9a35-204dcd3d4550-kube-api-access-qxxf4" pod "38924c59-63d4-40d5-9a35-204dcd3d4550" (UID: "38924c59-63d4-40d5-9a35-204dcd3d4550"). InnerVolumeSpecName "kube-api-access-qxxf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:01.040491 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:01.040474    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38924c59-63d4-40d5-9a35-204dcd3d4550-maas-api-xks-ca" pod "38924c59-63d4-40d5-9a35-204dcd3d4550" (UID: "38924c59-63d4-40d5-9a35-204dcd3d4550"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:06:01.140979 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:01.140940    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38924c59-63d4-40d5-9a35-204dcd3d4550-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:01.140979 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:01.140974    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxxf4\" (UniqueName: \"kubernetes.io/projected/38924c59-63d4-40d5-9a35-204dcd3d4550-kube-api-access-qxxf4\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:01.728340 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:01.728303    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:01.841505 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:01.841467    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8" event={"ID":"38924c59-63d4-40d5-9a35-204dcd3d4550","Type":"ContainerDied","Data":"c85e6656fbe0a4e399f58ea14060254ffa8765234c30f79f4e8c5d7f41aa8cc1"}
Oct 30 19:06:01.841505 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:01.841498    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8"
Oct 30 19:06:01.841780 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:01.841527    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c85e6656fbe0a4e399f58ea14060254ffa8765234c30f79f4e8c5d7f41aa8cc1"
Oct 30 19:06:03.405669 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:03.405635    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:04.056802 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:04.056759    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2953e0a1-t4l4q"]
Oct 30 19:06:04.096956 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:04.096920    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2953e0a1-t4l4q"]
Oct 30 19:06:04.841794 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:04.841755    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d" path="/var/lib/kubelet/pods/d7d7764c-082a-4bd9-aaf2-f1782bdf2a3d/volumes"
Oct 30 19:06:04.862094 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:04.862046    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c736cb70-frgvv"]
Oct 30 19:06:04.863281 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:04.863257    4924 cpu_manager.go:414] "containerMap: removing container" podUID="38924c59-63d4-40d5-9a35-204dcd3d4550" containerName="revoke-keys"
Oct 30 19:06:04.863416 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:04.863283    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38924c59-63d4-40d5-9a35-204dcd3d4550" containerName="revoke-keys"
Oct 30 19:06:04.863523 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:04.863496    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="38924c59-63d4-40d5-9a35-204dcd3d4550" containerName="revoke-keys"
Oct 30 19:06:04.868488 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:04.868465    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c736cb70-frgvv"
Oct 30 19:06:04.921132 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:04.921097    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c736cb70-frgvv"]
Oct 30 19:06:04.959200 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:04.959170    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:06:04.959417 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:04.959232    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:06:04.964205 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:04.964183    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:06:04.964205 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:04.964200    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:06:05.020388 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:05.020344    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j64f8\" (UniqueName: \"kubernetes.io/projected/cd776f26-6b23-44f8-be02-d58fa7960edf-kube-api-access-j64f8\") pod \"maas-api-revoke-sub-c736cb70-frgvv\" (UID: \"cd776f26-6b23-44f8-be02-d58fa7960edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c736cb70-frgvv"
Oct 30 19:06:05.020626 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:05.020599    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd776f26-6b23-44f8-be02-d58fa7960edf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c736cb70-frgvv\" (UID: \"cd776f26-6b23-44f8-be02-d58fa7960edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c736cb70-frgvv"
Oct 30 19:06:05.020792 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:05.020768    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd776f26-6b23-44f8-be02-d58fa7960edf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c736cb70-frgvv\" (UID: \"cd776f26-6b23-44f8-be02-d58fa7960edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c736cb70-frgvv"
Oct 30 19:06:05.122555 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:05.122518    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd776f26-6b23-44f8-be02-d58fa7960edf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c736cb70-frgvv\" (UID: \"cd776f26-6b23-44f8-be02-d58fa7960edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c736cb70-frgvv"
Oct 30 19:06:05.122745 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:05.122636    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd776f26-6b23-44f8-be02-d58fa7960edf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c736cb70-frgvv\" (UID: \"cd776f26-6b23-44f8-be02-d58fa7960edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c736cb70-frgvv"
Oct 30 19:06:05.122865 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:05.122842    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j64f8\" (UniqueName: \"kubernetes.io/projected/cd776f26-6b23-44f8-be02-d58fa7960edf-kube-api-access-j64f8\") pod \"maas-api-revoke-sub-c736cb70-frgvv\" (UID: \"cd776f26-6b23-44f8-be02-d58fa7960edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c736cb70-frgvv"
Oct 30 19:06:05.123490 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:05.123410    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd776f26-6b23-44f8-be02-d58fa7960edf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c736cb70-frgvv\" (UID: \"cd776f26-6b23-44f8-be02-d58fa7960edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c736cb70-frgvv"
Oct 30 19:06:05.125892 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:05.125866    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd776f26-6b23-44f8-be02-d58fa7960edf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c736cb70-frgvv\" (UID: \"cd776f26-6b23-44f8-be02-d58fa7960edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c736cb70-frgvv"
Oct 30 19:06:05.178781 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:05.178738    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j64f8\" (UniqueName: \"kubernetes.io/projected/cd776f26-6b23-44f8-be02-d58fa7960edf-kube-api-access-j64f8\") pod \"maas-api-revoke-sub-c736cb70-frgvv\" (UID: \"cd776f26-6b23-44f8-be02-d58fa7960edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c736cb70-frgvv"
Oct 30 19:06:05.182418 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:05.182389    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c736cb70-frgvv"
Oct 30 19:06:05.568720 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:06:05.568685    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd776f26_6b23_44f8_be02_d58fa7960edf.slice/crio-1f717a54e781588774e657e39db63911934dce23a63944258402a9bc23bb856f WatchSource:0}: Error finding container 1f717a54e781588774e657e39db63911934dce23a63944258402a9bc23bb856f: Status 404 returned error can't find the container with id 1f717a54e781588774e657e39db63911934dce23a63944258402a9bc23bb856f
Oct 30 19:06:05.570308 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:05.570280    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c736cb70-frgvv"]
Oct 30 19:06:05.882567 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:05.882534    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c736cb70-frgvv" event={"ID":"cd776f26-6b23-44f8-be02-d58fa7960edf","Type":"ContainerStarted","Data":"1f717a54e781588774e657e39db63911934dce23a63944258402a9bc23bb856f"}
Oct 30 19:06:06.288211 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.288167    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"]
Oct 30 19:06:06.292893 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.292876    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.398199 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.398164    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"]
Oct 30 19:06:06.444565 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.444529    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7e180c-7647-4b96-be77-c7d913ae4270-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.444744 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.444696    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7e180c-7647-4b96-be77-c7d913ae4270-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.444744 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.444728    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7e180c-7647-4b96-be77-c7d913ae4270-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.444862 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.444845    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9n67\" (UniqueName: \"kubernetes.io/projected/db7e180c-7647-4b96-be77-c7d913ae4270-kube-api-access-w9n67\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.444896 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.444878    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7e180c-7647-4b96-be77-c7d913ae4270-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.444953 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.444941    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7e180c-7647-4b96-be77-c7d913ae4270-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.444994 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.444975    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7e180c-7647-4b96-be77-c7d913ae4270-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.445143 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.445057    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7e180c-7647-4b96-be77-c7d913ae4270-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.445198 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.445174    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7e180c-7647-4b96-be77-c7d913ae4270-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.547012 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.546916    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7e180c-7647-4b96-be77-c7d913ae4270-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.547012 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.546978    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7e180c-7647-4b96-be77-c7d913ae4270-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.547256 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.547021    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7e180c-7647-4b96-be77-c7d913ae4270-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.547256 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.547083    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7e180c-7647-4b96-be77-c7d913ae4270-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.547364 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.547261    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7e180c-7647-4b96-be77-c7d913ae4270-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.547419 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.547384    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db7e180c-7647-4b96-be77-c7d913ae4270-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.547419 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.547411    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7e180c-7647-4b96-be77-c7d913ae4270-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.547526 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.547455    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7e180c-7647-4b96-be77-c7d913ae4270-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.547604 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.547586    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9n67\" (UniqueName: \"kubernetes.io/projected/db7e180c-7647-4b96-be77-c7d913ae4270-kube-api-access-w9n67\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.547668 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.547631    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7e180c-7647-4b96-be77-c7d913ae4270-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.547668 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.547633    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db7e180c-7647-4b96-be77-c7d913ae4270-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.547775 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.547730    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7e180c-7647-4b96-be77-c7d913ae4270-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.547909 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.547797    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db7e180c-7647-4b96-be77-c7d913ae4270-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.548031 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.547984    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db7e180c-7647-4b96-be77-c7d913ae4270-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.549728 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.549703    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db7e180c-7647-4b96-be77-c7d913ae4270-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.550016 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.550000    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db7e180c-7647-4b96-be77-c7d913ae4270-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.602698 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.602667    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db7e180c-7647-4b96-be77-c7d913ae4270-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.602819 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.602736    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9n67\" (UniqueName: \"kubernetes.io/projected/db7e180c-7647-4b96-be77-c7d913ae4270-kube-api-access-w9n67\") pod \"maas-default-gateway-openshift-default-75d7c785f-4kcpg\" (UID: \"db7e180c-7647-4b96-be77-c7d913ae4270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.609629 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.609605    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:06.869195 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.869114    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:06.933722 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.933683    4924 generic.go:356] "Generic (PLEG): container finished" podID="cd776f26-6b23-44f8-be02-d58fa7960edf" containerID="0f427cd2455c0cea4f3d0b5b0ded3c7f74a708e9f6bbba1a12f58d8ea8272b4e" exitCode=0
Oct 30 19:06:06.934264 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:06.933950    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c736cb70-frgvv" event={"ID":"cd776f26-6b23-44f8-be02-d58fa7960edf","Type":"ContainerDied","Data":"0f427cd2455c0cea4f3d0b5b0ded3c7f74a708e9f6bbba1a12f58d8ea8272b4e"}
Oct 30 19:06:07.090167 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:07.090134    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"]
Oct 30 19:06:07.092868 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:06:07.092837    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb7e180c_7647_4b96_be77_c7d913ae4270.slice/crio-22feecb3676865a5c20a978840358a13e5602b8d697e96dc793dbac0f51840f1 WatchSource:0}: Error finding container 22feecb3676865a5c20a978840358a13e5602b8d697e96dc793dbac0f51840f1: Status 404 returned error can't find the container with id 22feecb3676865a5c20a978840358a13e5602b8d697e96dc793dbac0f51840f1
Oct 30 19:06:07.096255 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:07.096224    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:06:07.096361 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:07.096290    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:06:07.096361 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:07.096320    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:06:07.161675 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:07.161629    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:07.343393 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:07.343349    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:07.944923 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:07.944887    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg" event={"ID":"db7e180c-7647-4b96-be77-c7d913ae4270","Type":"ContainerStarted","Data":"9251b274a05f93c53e5c8d9c23f314f74808df4ff4b0259716096934f2e0229b"}
Oct 30 19:06:07.945369 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:07.944947    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg" event={"ID":"db7e180c-7647-4b96-be77-c7d913ae4270","Type":"ContainerStarted","Data":"22feecb3676865a5c20a978840358a13e5602b8d697e96dc793dbac0f51840f1"}
Oct 30 19:06:08.068008 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:08.067944    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg" podStartSLOduration=2.067923845 podStartE2EDuration="2.067923845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:08.059886084 +0000 UTC m=+1769.336936540" watchObservedRunningTime="2026-10-30 19:06:08.067923845 +0000 UTC m=+1769.344974291"
Oct 30 19:06:08.198662 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:08.194766    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c736cb70-frgvv"
Oct 30 19:06:08.279825 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:08.279756    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd776f26-6b23-44f8-be02-d58fa7960edf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd776f26-6b23-44f8-be02-d58fa7960edf-maas-api-xks-ca\") pod \"cd776f26-6b23-44f8-be02-d58fa7960edf\" (UID: \"cd776f26-6b23-44f8-be02-d58fa7960edf\") "
Oct 30 19:06:08.280017 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:08.279934    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd776f26-6b23-44f8-be02-d58fa7960edf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd776f26-6b23-44f8-be02-d58fa7960edf-maas-api-openshift-ca\") pod \"cd776f26-6b23-44f8-be02-d58fa7960edf\" (UID: \"cd776f26-6b23-44f8-be02-d58fa7960edf\") "
Oct 30 19:06:08.280134 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:08.280118    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd776f26-6b23-44f8-be02-d58fa7960edf-kube-api-access-j64f8\" (UniqueName: \"kubernetes.io/projected/cd776f26-6b23-44f8-be02-d58fa7960edf-kube-api-access-j64f8\") pod \"cd776f26-6b23-44f8-be02-d58fa7960edf\" (UID: \"cd776f26-6b23-44f8-be02-d58fa7960edf\") "
Oct 30 19:06:08.280380 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:08.280355    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd776f26-6b23-44f8-be02-d58fa7960edf-maas-api-openshift-ca" pod "cd776f26-6b23-44f8-be02-d58fa7960edf" (UID: "cd776f26-6b23-44f8-be02-d58fa7960edf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:08.280819 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:08.280798    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd776f26-6b23-44f8-be02-d58fa7960edf-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:08.282060 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:08.282030    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd776f26-6b23-44f8-be02-d58fa7960edf-maas-api-xks-ca" pod "cd776f26-6b23-44f8-be02-d58fa7960edf" (UID: "cd776f26-6b23-44f8-be02-d58fa7960edf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:06:08.282317 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:08.282299    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd776f26-6b23-44f8-be02-d58fa7960edf-kube-api-access-j64f8" pod "cd776f26-6b23-44f8-be02-d58fa7960edf" (UID: "cd776f26-6b23-44f8-be02-d58fa7960edf"). InnerVolumeSpecName "kube-api-access-j64f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:08.383107 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:08.383054    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j64f8\" (UniqueName: \"kubernetes.io/projected/cd776f26-6b23-44f8-be02-d58fa7960edf-kube-api-access-j64f8\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:08.383107 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:08.383099    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd776f26-6b23-44f8-be02-d58fa7960edf-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:08.625086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:08.615906    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:08.625086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:08.616042    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-4kcpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 19:06:08.625086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:08.616113    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg" podUID="db7e180c-7647-4b96-be77-c7d913ae4270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 19:06:08.955055 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:08.955023    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c736cb70-frgvv"
Oct 30 19:06:08.955055 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:08.955053    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c736cb70-frgvv" event={"ID":"cd776f26-6b23-44f8-be02-d58fa7960edf","Type":"ContainerDied","Data":"1f717a54e781588774e657e39db63911934dce23a63944258402a9bc23bb856f"}
Oct 30 19:06:08.955492 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:08.955115    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f717a54e781588774e657e39db63911934dce23a63944258402a9bc23bb856f"
Oct 30 19:06:09.130911 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:09.130826    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c"]
Oct 30 19:06:09.239707 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:09.239674    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9"]
Oct 30 19:06:09.567657 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:09.567566    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:09.636572 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:09.634439    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-4kcpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 19:06:09.636572 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:09.634499    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg" podUID="db7e180c-7647-4b96-be77-c7d913ae4270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 19:06:09.769957 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:09.769893    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:10.641105 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:10.640100    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-4kcpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 19:06:10.641105 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:10.640162    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg" podUID="db7e180c-7647-4b96-be77-c7d913ae4270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 19:06:11.621089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:11.619786    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-4kcpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 19:06:11.621089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:11.619856    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg" podUID="db7e180c-7647-4b96-be77-c7d913ae4270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 19:06:12.054267 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:12.052996    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:12.260755 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:12.260700    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:12.627091 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:12.617635    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-4kcpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 19:06:12.627091 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:12.617696    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg" podUID="db7e180c-7647-4b96-be77-c7d913ae4270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 19:06:13.610560 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:13.610518    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-4kcpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 19:06:13.611115 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:13.610577    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg" podUID="db7e180c-7647-4b96-be77-c7d913ae4270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 19:06:14.131528 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:14.131482    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c" podUID="589c5535-35b9-429d-bce9-34ad1ca25a7c" containerName="payload-processing" containerID="cri-o://548e8daea73606d849dbdab211414d59728accc3518671dd58553711b35c05e1" gracePeriod=25
Oct 30 19:06:14.240930 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:14.240852    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9" podUID="02fb175b-4f55-480a-a3d0-6ea00d85adaa" containerName="payload-pre-processing" containerID="cri-o://58009424e7076fdc7807bfacec8312822cd202104c89b37792869e97c5cf8884" gracePeriod=25
Oct 30 19:06:14.260353 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:14.260269    4924 generic.go:356] "Generic (PLEG): container finished" podID="589c5535-35b9-429d-bce9-34ad1ca25a7c" containerID="548e8daea73606d849dbdab211414d59728accc3518671dd58553711b35c05e1" exitCode=0
Oct 30 19:06:14.260353 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:14.260320    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c" event={"ID":"589c5535-35b9-429d-bce9-34ad1ca25a7c","Type":"ContainerDied","Data":"548e8daea73606d849dbdab211414d59728accc3518671dd58553711b35c05e1"}
Oct 30 19:06:14.462618 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:14.462575    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c"
Oct 30 19:06:14.472521 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:14.472489    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/589c5535-35b9-429d-bce9-34ad1ca25a7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589c5535-35b9-429d-bce9-34ad1ca25a7c-plugins-config-volume\") pod \"589c5535-35b9-429d-bce9-34ad1ca25a7c\" (UID: \"589c5535-35b9-429d-bce9-34ad1ca25a7c\") "
Oct 30 19:06:14.472780 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:14.472755    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/589c5535-35b9-429d-bce9-34ad1ca25a7c-kube-api-access-kwxcg\" (UniqueName: \"kubernetes.io/projected/589c5535-35b9-429d-bce9-34ad1ca25a7c-kube-api-access-kwxcg\") pod \"589c5535-35b9-429d-bce9-34ad1ca25a7c\" (UID: \"589c5535-35b9-429d-bce9-34ad1ca25a7c\") "
Oct 30 19:06:14.473185 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:14.473130    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/589c5535-35b9-429d-bce9-34ad1ca25a7c-plugins-config-volume" pod "589c5535-35b9-429d-bce9-34ad1ca25a7c" (UID: "589c5535-35b9-429d-bce9-34ad1ca25a7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:14.473509 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:14.473483    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589c5535-35b9-429d-bce9-34ad1ca25a7c-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:14.478315 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:14.478282    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/589c5535-35b9-429d-bce9-34ad1ca25a7c-kube-api-access-kwxcg" pod "589c5535-35b9-429d-bce9-34ad1ca25a7c" (UID: "589c5535-35b9-429d-bce9-34ad1ca25a7c"). InnerVolumeSpecName "kube-api-access-kwxcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:14.554205 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:14.554175    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9"
Oct 30 19:06:14.575226 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:14.574627    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02fb175b-4f55-480a-a3d0-6ea00d85adaa-kube-api-access-8m45b\" (UniqueName: \"kubernetes.io/projected/02fb175b-4f55-480a-a3d0-6ea00d85adaa-kube-api-access-8m45b\") pod \"02fb175b-4f55-480a-a3d0-6ea00d85adaa\" (UID: \"02fb175b-4f55-480a-a3d0-6ea00d85adaa\") "
Oct 30 19:06:14.575226 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:14.574777    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02fb175b-4f55-480a-a3d0-6ea00d85adaa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fb175b-4f55-480a-a3d0-6ea00d85adaa-plugins-config-volume\") pod \"02fb175b-4f55-480a-a3d0-6ea00d85adaa\" (UID: \"02fb175b-4f55-480a-a3d0-6ea00d85adaa\") "
Oct 30 19:06:14.577204 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:14.575584    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwxcg\" (UniqueName: \"kubernetes.io/projected/589c5535-35b9-429d-bce9-34ad1ca25a7c-kube-api-access-kwxcg\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:14.577204 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:14.575932    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02fb175b-4f55-480a-a3d0-6ea00d85adaa-plugins-config-volume" pod "02fb175b-4f55-480a-a3d0-6ea00d85adaa" (UID: "02fb175b-4f55-480a-a3d0-6ea00d85adaa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:14.577808 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:14.577785    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02fb175b-4f55-480a-a3d0-6ea00d85adaa-kube-api-access-8m45b" pod "02fb175b-4f55-480a-a3d0-6ea00d85adaa" (UID: "02fb175b-4f55-480a-a3d0-6ea00d85adaa"). InnerVolumeSpecName "kube-api-access-8m45b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:14.610677 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:14.610650    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-4kcpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 30 19:06:14.611099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:14.610701    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg" podUID="db7e180c-7647-4b96-be77-c7d913ae4270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 30 19:06:14.677720 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:14.677645    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m45b\" (UniqueName: \"kubernetes.io/projected/02fb175b-4f55-480a-a3d0-6ea00d85adaa-kube-api-access-8m45b\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:14.677720 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:14.677674    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02fb175b-4f55-480a-a3d0-6ea00d85adaa-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:15.049441 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:15.049344    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eb09cf8-mcsfw"]
Oct 30 19:06:15.053866 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:15.053840    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eb09cf8-mcsfw"]
Oct 30 19:06:15.271650 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:15.271618    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c"
Oct 30 19:06:15.271848 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:15.271656    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c" event={"ID":"589c5535-35b9-429d-bce9-34ad1ca25a7c","Type":"ContainerDied","Data":"0b0f64ff62103880fe17782b1e3e9ee5a69e28d62eba66d66079a34bf1ab3849"}
Oct 30 19:06:15.271848 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:15.271721    4924 scope.go:122] "RemoveContainer" containerID="548e8daea73606d849dbdab211414d59728accc3518671dd58553711b35c05e1"
Oct 30 19:06:15.273769 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:15.273742    4924 generic.go:356] "Generic (PLEG): container finished" podID="02fb175b-4f55-480a-a3d0-6ea00d85adaa" containerID="58009424e7076fdc7807bfacec8312822cd202104c89b37792869e97c5cf8884" exitCode=0
Oct 30 19:06:15.273913 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:15.273813    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9"
Oct 30 19:06:15.273913 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:15.273844    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9" event={"ID":"02fb175b-4f55-480a-a3d0-6ea00d85adaa","Type":"ContainerDied","Data":"58009424e7076fdc7807bfacec8312822cd202104c89b37792869e97c5cf8884"}
Oct 30 19:06:15.273913 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:15.273889    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9" event={"ID":"02fb175b-4f55-480a-a3d0-6ea00d85adaa","Type":"ContainerDied","Data":"fbde321f05be8aef11b8897b1caa801f3fbd6ce09a7127226208543bebfb7f5b"}
Oct 30 19:06:15.285474 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:15.285454    4924 scope.go:122] "RemoveContainer" containerID="58009424e7076fdc7807bfacec8312822cd202104c89b37792869e97c5cf8884"
Oct 30 19:06:15.297213 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:15.297196    4924 scope.go:122] "RemoveContainer" containerID="58009424e7076fdc7807bfacec8312822cd202104c89b37792869e97c5cf8884"
Oct 30 19:06:15.297523 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:06:15.297484    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58009424e7076fdc7807bfacec8312822cd202104c89b37792869e97c5cf8884\": container with ID starting with 58009424e7076fdc7807bfacec8312822cd202104c89b37792869e97c5cf8884 not found: ID does not exist" containerID="58009424e7076fdc7807bfacec8312822cd202104c89b37792869e97c5cf8884"
Oct 30 19:06:15.297643 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:15.297528    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58009424e7076fdc7807bfacec8312822cd202104c89b37792869e97c5cf8884"} err="failed to get container status \"58009424e7076fdc7807bfacec8312822cd202104c89b37792869e97c5cf8884\": rpc error: code = NotFound desc = could not find container \"58009424e7076fdc7807bfacec8312822cd202104c89b37792869e97c5cf8884\": container with ID starting with 58009424e7076fdc7807bfacec8312822cd202104c89b37792869e97c5cf8884 not found: ID does not exist"
Oct 30 19:06:15.432849 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:15.432817    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9"]
Oct 30 19:06:15.437280 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:15.437252    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06203ab6-77cc6c5499-hbsw9"]
Oct 30 19:06:15.530690 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:15.530650    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:15.539267 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:15.539112    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c"]
Oct 30 19:06:15.543906 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:15.543863    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06203ab6-6c978994bd-fvd4c"]
Oct 30 19:06:15.712429 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:06:15.712349    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:06:16.025274 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:16.025200    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:16.025619 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:16.025592    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:16.026314 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:16.026286    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg"
Oct 30 19:06:16.836607 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:16.836570    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02fb175b-4f55-480a-a3d0-6ea00d85adaa" path="/var/lib/kubelet/pods/02fb175b-4f55-480a-a3d0-6ea00d85adaa/volumes"
Oct 30 19:06:16.837101 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:16.837084    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="589c5535-35b9-429d-bce9-34ad1ca25a7c" path="/var/lib/kubelet/pods/589c5535-35b9-429d-bce9-34ad1ca25a7c/volumes"
Oct 30 19:06:16.837550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:16.837536    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef7c3720-9c21-4ba3-b919-44f1009889b4" path="/var/lib/kubelet/pods/ef7c3720-9c21-4ba3-b919-44f1009889b4/volumes"
Oct 30 19:06:17.093722 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:17.093624    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:21.812222 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:21.812186    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:22.013167 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:22.010524    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:26.001727 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:26.001694    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:27.619105 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:27.619052    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:27.871224 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:27.871137    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:28.082090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:28.076834    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:28.175077 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:28.174967    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8"]
Oct 30 19:06:28.221392 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:28.221355    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06203ab6-vbrd8"]
Oct 30 19:06:28.625198 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:28.625161    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:28.836491 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:28.836457    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38924c59-63d4-40d5-9a35-204dcd3d4550" path="/var/lib/kubelet/pods/38924c59-63d4-40d5-9a35-204dcd3d4550/volumes"
Oct 30 19:06:33.787471 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:33.787437    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:34.071381 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:34.071226    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:34.262185 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:34.262156    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:34.971276 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:34.971242    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:35.172224 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:35.172192    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:35.276143 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:35.276016    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"]
Oct 30 19:06:35.277732 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:35.277696    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9" podUID="61957487-4e2c-47d7-9977-1d34f3c2534f" containerName="istio-proxy" containerID="cri-o://ada40b7f7abe844ddb41950a27d93d871473b4ed9aac378e4cc8cb1730245b9a" gracePeriod=30
Oct 30 19:06:35.451095 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:35.445152    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:35.832632 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:35.832600    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:36.031443 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:36.031412    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:36.311694 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:36.311643    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:37.148563 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:37.148526    4924 patch_prober.go:29] interesting pod/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:06:37.149033 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:37.148599    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9" podUID="61957487-4e2c-47d7-9977-1d34f3c2534f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:06:38.282055 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:38.281990    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:39.110362 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:39.110331    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 19:06:39.124917 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:39.124891    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 19:06:39.785869 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:39.785830    4924 scope.go:122] "RemoveContainer" containerID="807e68e10c71ab16adfaf75f8d14628b323478ccf5890a815442b3360b09fa64"
Oct 30 19:06:39.796644 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:39.796624    4924 scope.go:122] "RemoveContainer" containerID="85beb229203eaba8d32dfd0642927f00f25a3aaf7da9f0ed90168f7ba32c5563"
Oct 30 19:06:39.806048 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:39.806019    4924 scope.go:122] "RemoveContainer" containerID="1bf571168ccdcb044f03f5555781cb8a22171b72213d30611995c250c7f2dc0d"
Oct 30 19:06:39.815385 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:39.815368    4924 scope.go:122] "RemoveContainer" containerID="61b010d9e4fbc4c8c6b532645b547ba786c706b0a3af185c5e8c55b89b6ee4e9"
Oct 30 19:06:39.824765 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:39.824745    4924 scope.go:122] "RemoveContainer" containerID="bbad9d78ce4f774c2b601174d2ac179b21b9de2af5a238e9ec77fefdb5d0dfcb"
Oct 30 19:06:39.834357 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:39.834341    4924 scope.go:122] "RemoveContainer" containerID="dbf7cefba4e8666eef1fb7c3d45d8896e9f4b69d159ba1ac9b2854eab135c002"
Oct 30 19:06:39.844282 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:39.844261    4924 scope.go:122] "RemoveContainer" containerID="3b3a02295eaadb323161ea4e7a02e3ab549df6ec3b0130a441226ee632370885"
Oct 30 19:06:39.853500 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:39.853480    4924 scope.go:122] "RemoveContainer" containerID="4937402652d37cc13ed384b6f0f927046b9a277ffb387470dd75b9492536ab33"
Oct 30 19:06:39.863200 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:39.863185    4924 scope.go:122] "RemoveContainer" containerID="22ea07c211abac5dbf801704eadca516cf07ff14030f54cd706af40931d94c20"
Oct 30 19:06:39.968200 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:39.968168    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:40.595608 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.595571    4924 generic.go:356] "Generic (PLEG): container finished" podID="61957487-4e2c-47d7-9977-1d34f3c2534f" containerID="ada40b7f7abe844ddb41950a27d93d871473b4ed9aac378e4cc8cb1730245b9a" exitCode=0
Oct 30 19:06:40.595743 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.595657    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9" event={"ID":"61957487-4e2c-47d7-9977-1d34f3c2534f","Type":"ContainerDied","Data":"ada40b7f7abe844ddb41950a27d93d871473b4ed9aac378e4cc8cb1730245b9a"}
Oct 30 19:06:40.595819 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.595750    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9" event={"ID":"61957487-4e2c-47d7-9977-1d34f3c2534f","Type":"ContainerDied","Data":"c51e01b837132b568d43541aa4b8c38e9b4d76daf4798ad2157b73e5293ffd7d"}
Oct 30 19:06:40.595819 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.595800    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c51e01b837132b568d43541aa4b8c38e9b4d76daf4798ad2157b73e5293ffd7d"
Oct 30 19:06:40.595942 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.595818    4924 scope.go:122] "RemoveContainer" containerID="24cf43daf4b82557e9a952375329352223416944d7697fc9e9ea03026a2c27ea"
Oct 30 19:06:40.624235 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.624208    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:06:40.754601 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.754565    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-token\" (UniqueName: \"kubernetes.io/projected/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-token\") pod \"61957487-4e2c-47d7-9977-1d34f3c2534f\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") "
Oct 30 19:06:40.754805 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.754747    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-credential-socket\") pod \"61957487-4e2c-47d7-9977-1d34f3c2534f\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") "
Oct 30 19:06:40.754805 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.754789    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-workload-socket\") pod \"61957487-4e2c-47d7-9977-1d34f3c2534f\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") "
Oct 30 19:06:40.754935 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.754871    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-workload-certs\") pod \"61957487-4e2c-47d7-9977-1d34f3c2534f\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") "
Oct 30 19:06:40.754988 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.754963    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-podinfo\") pod \"61957487-4e2c-47d7-9977-1d34f3c2534f\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") "
Oct 30 19:06:40.755342 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.755083    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-credential-socket" pod "61957487-4e2c-47d7-9977-1d34f3c2534f" (UID: "61957487-4e2c-47d7-9977-1d34f3c2534f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:40.755342 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.755105    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61957487-4e2c-47d7-9977-1d34f3c2534f-kube-api-access-fstj2\" (UniqueName: \"kubernetes.io/projected/61957487-4e2c-47d7-9977-1d34f3c2534f-kube-api-access-fstj2\") pod \"61957487-4e2c-47d7-9977-1d34f3c2534f\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") "
Oct 30 19:06:40.755342 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.755202    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-workload-certs" pod "61957487-4e2c-47d7-9977-1d34f3c2534f" (UID: "61957487-4e2c-47d7-9977-1d34f3c2534f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:40.755342 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.755209    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-envoy\") pod \"61957487-4e2c-47d7-9977-1d34f3c2534f\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") "
Oct 30 19:06:40.755698 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.755342    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-data\") pod \"61957487-4e2c-47d7-9977-1d34f3c2534f\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") "
Oct 30 19:06:40.755698 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.755387    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61957487-4e2c-47d7-9977-1d34f3c2534f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61957487-4e2c-47d7-9977-1d34f3c2534f-istiod-ca-cert\") pod \"61957487-4e2c-47d7-9977-1d34f3c2534f\" (UID: \"61957487-4e2c-47d7-9977-1d34f3c2534f\") "
Oct 30 19:06:40.755698 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.755664    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-data" pod "61957487-4e2c-47d7-9977-1d34f3c2534f" (UID: "61957487-4e2c-47d7-9977-1d34f3c2534f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:40.756327 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.756301    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:40.756448 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.756331    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:40.756448 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.756403    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:40.756650 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.756630    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-workload-socket" pod "61957487-4e2c-47d7-9977-1d34f3c2534f" (UID: "61957487-4e2c-47d7-9977-1d34f3c2534f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:40.756758 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.756727    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61957487-4e2c-47d7-9977-1d34f3c2534f-istiod-ca-cert" pod "61957487-4e2c-47d7-9977-1d34f3c2534f" (UID: "61957487-4e2c-47d7-9977-1d34f3c2534f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:40.757766 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.757741    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-envoy" pod "61957487-4e2c-47d7-9977-1d34f3c2534f" (UID: "61957487-4e2c-47d7-9977-1d34f3c2534f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:40.757988 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.757947    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-token" pod "61957487-4e2c-47d7-9977-1d34f3c2534f" (UID: "61957487-4e2c-47d7-9977-1d34f3c2534f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:40.758368 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.758346    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-podinfo" pod "61957487-4e2c-47d7-9977-1d34f3c2534f" (UID: "61957487-4e2c-47d7-9977-1d34f3c2534f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:06:40.759647 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.759623    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61957487-4e2c-47d7-9977-1d34f3c2534f-kube-api-access-fstj2" pod "61957487-4e2c-47d7-9977-1d34f3c2534f" (UID: "61957487-4e2c-47d7-9977-1d34f3c2534f"). InnerVolumeSpecName "kube-api-access-fstj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:40.862437 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.861385    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:40.862437 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.861424    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fstj2\" (UniqueName: \"kubernetes.io/projected/61957487-4e2c-47d7-9977-1d34f3c2534f-kube-api-access-fstj2\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:40.862437 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.861442    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:40.862437 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.861458    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61957487-4e2c-47d7-9977-1d34f3c2534f-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:40.862437 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.861471    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61957487-4e2c-47d7-9977-1d34f3c2534f-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:40.862437 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:40.861484    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61957487-4e2c-47d7-9977-1d34f3c2534f-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:41.605111 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:41.605081    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"
Oct 30 19:06:41.799089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:41.799041    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"]
Oct 30 19:06:41.845933 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:41.845899    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-06203ab6-openshift-default-7ddf46b7bf-9c8c9"]
Oct 30 19:06:42.836051 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:42.836020    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61957487-4e2c-47d7-9977-1d34f3c2534f" path="/var/lib/kubelet/pods/61957487-4e2c-47d7-9977-1d34f3c2534f/volumes"
Oct 30 19:06:42.993263 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:42.993233    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:43.187187 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:43.187158    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:43.565973 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:43.565893    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:44.474760 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:44.474219    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:44.936511 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:44.936472    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:45.215513 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:45.215415    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:45.713676 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:06:45.713643    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:06:49.510904 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:49.510865    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:49.715954 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:49.715925    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:49.912886 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:49.912858    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:51.633056 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.633018    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"]
Oct 30 19:06:51.633899 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.633879    4924 cpu_manager.go:414] "containerMap: removing container" podUID="cd776f26-6b23-44f8-be02-d58fa7960edf" containerName="revoke-keys"
Oct 30 19:06:51.633962 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.633900    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd776f26-6b23-44f8-be02-d58fa7960edf" containerName="revoke-keys"
Oct 30 19:06:51.633962 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.633923    4924 cpu_manager.go:414] "containerMap: removing container" podUID="61957487-4e2c-47d7-9977-1d34f3c2534f" containerName="istio-proxy"
Oct 30 19:06:51.633962 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.633928    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61957487-4e2c-47d7-9977-1d34f3c2534f" containerName="istio-proxy"
Oct 30 19:06:51.633962 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.633942    4924 cpu_manager.go:414] "containerMap: removing container" podUID="61957487-4e2c-47d7-9977-1d34f3c2534f" containerName="istio-proxy"
Oct 30 19:06:51.633962 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.633948    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61957487-4e2c-47d7-9977-1d34f3c2534f" containerName="istio-proxy"
Oct 30 19:06:51.634166 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.633970    4924 cpu_manager.go:414] "containerMap: removing container" podUID="02fb175b-4f55-480a-a3d0-6ea00d85adaa" containerName="payload-pre-processing"
Oct 30 19:06:51.634166 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.633976    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02fb175b-4f55-480a-a3d0-6ea00d85adaa" containerName="payload-pre-processing"
Oct 30 19:06:51.634166 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.633995    4924 cpu_manager.go:414] "containerMap: removing container" podUID="589c5535-35b9-429d-bce9-34ad1ca25a7c" containerName="payload-processing"
Oct 30 19:06:51.634166 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.634000    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="589c5535-35b9-429d-bce9-34ad1ca25a7c" containerName="payload-processing"
Oct 30 19:06:51.634166 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.634124    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="61957487-4e2c-47d7-9977-1d34f3c2534f" containerName="istio-proxy"
Oct 30 19:06:51.634166 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.634136    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="02fb175b-4f55-480a-a3d0-6ea00d85adaa" containerName="payload-pre-processing"
Oct 30 19:06:51.634166 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.634146    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="589c5535-35b9-429d-bce9-34ad1ca25a7c" containerName="payload-processing"
Oct 30 19:06:51.634166 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.634155    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd776f26-6b23-44f8-be02-d58fa7960edf" containerName="revoke-keys"
Oct 30 19:06:51.639161 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.639135    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.738042 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.738003    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c996cd-4196-4e8a-b067-9fc146104887-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.738270 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.738105    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75c996cd-4196-4e8a-b067-9fc146104887-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.738270 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.738149    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6p7s\" (UniqueName: \"kubernetes.io/projected/75c996cd-4196-4e8a-b067-9fc146104887-kube-api-access-s6p7s\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.738270 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.738212    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c996cd-4196-4e8a-b067-9fc146104887-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.738270 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.738253    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c996cd-4196-4e8a-b067-9fc146104887-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.738481 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.738303    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c996cd-4196-4e8a-b067-9fc146104887-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.738481 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.738400    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c996cd-4196-4e8a-b067-9fc146104887-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.738481 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.738469    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c996cd-4196-4e8a-b067-9fc146104887-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.738645 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.738579    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c996cd-4196-4e8a-b067-9fc146104887-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.741473 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.741440    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"]
Oct 30 19:06:51.839458 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.839428    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75c996cd-4196-4e8a-b067-9fc146104887-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.839638 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.839470    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6p7s\" (UniqueName: \"kubernetes.io/projected/75c996cd-4196-4e8a-b067-9fc146104887-kube-api-access-s6p7s\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.839638 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.839515    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c996cd-4196-4e8a-b067-9fc146104887-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.839638 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.839539    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c996cd-4196-4e8a-b067-9fc146104887-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.839638 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.839574    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c996cd-4196-4e8a-b067-9fc146104887-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.839886 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.839651    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c996cd-4196-4e8a-b067-9fc146104887-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.839886 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.839708    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c996cd-4196-4e8a-b067-9fc146104887-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.839886 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.839793    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c996cd-4196-4e8a-b067-9fc146104887-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.840049 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.839908    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c996cd-4196-4e8a-b067-9fc146104887-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.840049 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.839988    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c996cd-4196-4e8a-b067-9fc146104887-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.840049 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.840022    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c996cd-4196-4e8a-b067-9fc146104887-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.840232 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.840189    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75c996cd-4196-4e8a-b067-9fc146104887-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.840358 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.840331    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75c996cd-4196-4e8a-b067-9fc146104887-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.840479 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.840403    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75c996cd-4196-4e8a-b067-9fc146104887-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.842245 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.842218    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75c996cd-4196-4e8a-b067-9fc146104887-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.842488 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.842468    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75c996cd-4196-4e8a-b067-9fc146104887-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.895936 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.895859    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6p7s\" (UniqueName: \"kubernetes.io/projected/75c996cd-4196-4e8a-b067-9fc146104887-kube-api-access-s6p7s\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.896617 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.896589    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75c996cd-4196-4e8a-b067-9fc146104887-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-jvm6x\" (UID: \"75c996cd-4196-4e8a-b067-9fc146104887\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:51.952853 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:51.952824    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:52.380098 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:06:52.380033    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c996cd_4196_4e8a_b067_9fc146104887.slice/crio-70a6063e6c2990b348887a916028f8b193d73b4a169398edfc6013fd969e0300 WatchSource:0}: Error finding container 70a6063e6c2990b348887a916028f8b193d73b4a169398edfc6013fd969e0300: Status 404 returned error can't find the container with id 70a6063e6c2990b348887a916028f8b193d73b4a169398edfc6013fd969e0300
Oct 30 19:06:52.380565 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:52.380541    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"]
Oct 30 19:06:52.383096 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:52.383049    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:06:52.383178 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:52.383133    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:06:52.383178 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:52.383162    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:06:52.626010 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:52.625928    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:52.832352 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:52.832287    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:52.858252 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:52.857869    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x" event={"ID":"75c996cd-4196-4e8a-b067-9fc146104887","Type":"ContainerStarted","Data":"6bb467ae53f93ff4d6f2bc482299ec759b64872d2ef0454f22769ae842e85266"}
Oct 30 19:06:52.858252 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:52.857917    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x" event={"ID":"75c996cd-4196-4e8a-b067-9fc146104887","Type":"ContainerStarted","Data":"70a6063e6c2990b348887a916028f8b193d73b4a169398edfc6013fd969e0300"}
Oct 30 19:06:52.949347 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:52.949226    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x" podStartSLOduration=1.949206265 podStartE2EDuration="1.949206265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:52.943263338 +0000 UTC m=+1814.220313783" watchObservedRunningTime="2026-10-30 19:06:52.949206265 +0000 UTC m=+1814.226256711"
Oct 30 19:06:52.953164 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:52.953127    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:06:53.954920 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:53.954890    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jvm6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 19:06:53.955324 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:53.954938    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x" podUID="75c996cd-4196-4e8a-b067-9fc146104887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 19:06:54.841473 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:54.841434    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:54.958117 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:54.953482    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jvm6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 19:06:54.958117 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:54.953549    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x" podUID="75c996cd-4196-4e8a-b067-9fc146104887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 19:06:55.954106 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:55.954037    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jvm6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 19:06:55.954343 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:55.954133    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x" podUID="75c996cd-4196-4e8a-b067-9fc146104887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 19:06:56.953341 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:56.953306    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jvm6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 19:06:56.953838 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:56.953358    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x" podUID="75c996cd-4196-4e8a-b067-9fc146104887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 19:06:57.953282 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:57.953247    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jvm6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 19:06:57.953507 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:57.953309    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x" podUID="75c996cd-4196-4e8a-b067-9fc146104887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 19:06:58.953375 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:58.953335    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jvm6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 19:06:58.953564 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:58.953401    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x" podUID="75c996cd-4196-4e8a-b067-9fc146104887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 19:06:59.899990 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:59.899942    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:06:59.953864 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:59.953824    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jvm6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 30 19:06:59.954430 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:06:59.953910    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x" podUID="75c996cd-4196-4e8a-b067-9fc146104887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 30 19:07:00.183769 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:00.183679    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:00.425368 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:00.425340    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:01.285912 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:01.285874    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:07:01.286503 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:01.286478    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:07:01.287232 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:01.287210    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x"
Oct 30 19:07:02.050406 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:02.050372    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e07bd9b4-k4rk5"]
Oct 30 19:07:02.055777 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:02.055744    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e07bd9b4-k4rk5"]
Oct 30 19:07:02.838309 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:02.838274    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9828d1fc-c17d-4cd4-947f-3ef34336be4a" path="/var/lib/kubelet/pods/9828d1fc-c17d-4cd4-947f-3ef34336be4a/volumes"
Oct 30 19:07:04.539724 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:04.539687    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:06.236467 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:06.236391    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:06.516283 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:06.516195    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:06.697360 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:06.697324    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cef683f-fvknj"]
Oct 30 19:07:06.735610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:06.735576    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4cef683f-fvknj"]
Oct 30 19:07:06.764252 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:06.764223    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:06.844552 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:06.844464    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cb370d9-c034-45eb-862c-20d8fe198847" path="/var/lib/kubelet/pods/2cb370d9-c034-45eb-862c-20d8fe198847/volumes"
Oct 30 19:07:06.901036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:06.900990    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"]
Oct 30 19:07:06.901292 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:06.901258    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn" podUID="77664ce4-7a08-4b33-bccf-2e0755f13be3" containerName="istio-proxy" containerID="cri-o://87deaaa6e396f6d73ce507587a68176b8ebca2bcf72f35e9ac7291063f496b35" gracePeriod=30
Oct 30 19:07:07.098699 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:07.098666    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:08.355983 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:08.355946    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:09.106208 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:09.106173    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:09.751119 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:09.751081    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:12.065991 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.065956    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:12.082134 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.080738    4924 generic.go:356] "Generic (PLEG): container finished" podID="77664ce4-7a08-4b33-bccf-2e0755f13be3" containerID="87deaaa6e396f6d73ce507587a68176b8ebca2bcf72f35e9ac7291063f496b35" exitCode=0
Oct 30 19:07:12.092483 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.087362    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn" event={"ID":"77664ce4-7a08-4b33-bccf-2e0755f13be3","Type":"ContainerDied","Data":"87deaaa6e396f6d73ce507587a68176b8ebca2bcf72f35e9ac7291063f496b35"}
Oct 30 19:07:12.092483 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.087456    4924 scope.go:122] "RemoveContainer" containerID="fcb1e7fd8d711d7a9fbbaf24aa04a394a144ac4ca60c43d5cadbf94b24d85766"
Oct 30 19:07:12.245865 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.245602    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:07:12.268634 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.268598    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-envoy\") pod \"77664ce4-7a08-4b33-bccf-2e0755f13be3\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") "
Oct 30 19:07:12.268821 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.268661    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-workload-certs\") pod \"77664ce4-7a08-4b33-bccf-2e0755f13be3\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") "
Oct 30 19:07:12.268821 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.268738    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-podinfo\") pod \"77664ce4-7a08-4b33-bccf-2e0755f13be3\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") "
Oct 30 19:07:12.268821 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.268769    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-credential-socket\") pod \"77664ce4-7a08-4b33-bccf-2e0755f13be3\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") "
Oct 30 19:07:12.268821 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.268810    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-data\") pod \"77664ce4-7a08-4b33-bccf-2e0755f13be3\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") "
Oct 30 19:07:12.269049 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.268955    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-workload-certs" pod "77664ce4-7a08-4b33-bccf-2e0755f13be3" (UID: "77664ce4-7a08-4b33-bccf-2e0755f13be3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:12.269133 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.269059    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77664ce4-7a08-4b33-bccf-2e0755f13be3-kube-api-access-tqn5l\" (UniqueName: \"kubernetes.io/projected/77664ce4-7a08-4b33-bccf-2e0755f13be3-kube-api-access-tqn5l\") pod \"77664ce4-7a08-4b33-bccf-2e0755f13be3\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") "
Oct 30 19:07:12.269188 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.269125    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-credential-socket" pod "77664ce4-7a08-4b33-bccf-2e0755f13be3" (UID: "77664ce4-7a08-4b33-bccf-2e0755f13be3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:12.269188 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.269129    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-token\" (UniqueName: \"kubernetes.io/projected/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-token\") pod \"77664ce4-7a08-4b33-bccf-2e0755f13be3\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") "
Oct 30 19:07:12.269188 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.269166    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-data" pod "77664ce4-7a08-4b33-bccf-2e0755f13be3" (UID: "77664ce4-7a08-4b33-bccf-2e0755f13be3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:12.269365 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.269189    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77664ce4-7a08-4b33-bccf-2e0755f13be3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77664ce4-7a08-4b33-bccf-2e0755f13be3-istiod-ca-cert\") pod \"77664ce4-7a08-4b33-bccf-2e0755f13be3\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") "
Oct 30 19:07:12.269365 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.269233    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-workload-socket\") pod \"77664ce4-7a08-4b33-bccf-2e0755f13be3\" (UID: \"77664ce4-7a08-4b33-bccf-2e0755f13be3\") "
Oct 30 19:07:12.269614 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.269591    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-workload-socket" pod "77664ce4-7a08-4b33-bccf-2e0755f13be3" (UID: "77664ce4-7a08-4b33-bccf-2e0755f13be3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:12.269772 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.269752    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77664ce4-7a08-4b33-bccf-2e0755f13be3-istiod-ca-cert" pod "77664ce4-7a08-4b33-bccf-2e0755f13be3" (UID: "77664ce4-7a08-4b33-bccf-2e0755f13be3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:12.270523 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.270498    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77664ce4-7a08-4b33-bccf-2e0755f13be3-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:12.270574 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.270534    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:12.270574 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.270550    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:12.270574 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.270566    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:12.270669 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.270583    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:12.271057 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.271037    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-envoy" pod "77664ce4-7a08-4b33-bccf-2e0755f13be3" (UID: "77664ce4-7a08-4b33-bccf-2e0755f13be3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:12.271436 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.271410    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-podinfo" pod "77664ce4-7a08-4b33-bccf-2e0755f13be3" (UID: "77664ce4-7a08-4b33-bccf-2e0755f13be3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:07:12.271524 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.271501    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-token" pod "77664ce4-7a08-4b33-bccf-2e0755f13be3" (UID: "77664ce4-7a08-4b33-bccf-2e0755f13be3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:12.271569 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.271521    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77664ce4-7a08-4b33-bccf-2e0755f13be3-kube-api-access-tqn5l" pod "77664ce4-7a08-4b33-bccf-2e0755f13be3" (UID: "77664ce4-7a08-4b33-bccf-2e0755f13be3"). InnerVolumeSpecName "kube-api-access-tqn5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:12.372285 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.372253    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqn5l\" (UniqueName: \"kubernetes.io/projected/77664ce4-7a08-4b33-bccf-2e0755f13be3-kube-api-access-tqn5l\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:12.372285 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.372282    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:12.372285 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.372294    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:12.372506 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.372303    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77664ce4-7a08-4b33-bccf-2e0755f13be3-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:12.866960 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.866925    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"]
Oct 30 19:07:12.867222 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:12.867191    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw" podUID="ccd81788-05eb-4696-938f-d6dc6d3ab2ce" containerName="istio-proxy" containerID="cri-o://a20ef174b15c0d5c8dd4912907e77576e9907973a6773c5a6bbb2f624b806b7b" gracePeriod=30
Oct 30 19:07:13.049197 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:13.049164    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb8901fc-z8s85"]
Oct 30 19:07:13.053385 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:13.053349    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb8901fc-z8s85"]
Oct 30 19:07:13.096774 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:13.096749    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"
Oct 30 19:07:13.097227 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:13.096746    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn" event={"ID":"77664ce4-7a08-4b33-bccf-2e0755f13be3","Type":"ContainerDied","Data":"c3480923532f4bbbb2a1fdc9b003d7b53d3f016ab18935fca5282fdaec828bbf"}
Oct 30 19:07:13.097227 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:13.096924    4924 scope.go:122] "RemoveContainer" containerID="87deaaa6e396f6d73ce507587a68176b8ebca2bcf72f35e9ac7291063f496b35"
Oct 30 19:07:13.257368 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:13.257326    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"]
Oct 30 19:07:13.261507 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:13.261452    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-4cef683f-openshift-default-7cc966f665-b4njn"]
Oct 30 19:07:13.951047 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:13.951017    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:14.161384 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:14.161353    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:14.837667 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:14.837629    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77664ce4-7a08-4b33-bccf-2e0755f13be3" path="/var/lib/kubelet/pods/77664ce4-7a08-4b33-bccf-2e0755f13be3/volumes"
Oct 30 19:07:14.838221 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:14.838204    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad056395-5148-4f95-97cc-69163d684a81" path="/var/lib/kubelet/pods/ad056395-5148-4f95-97cc-69163d684a81/volumes"
Oct 30 19:07:15.712050 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:07:15.712019    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:07:17.135053 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:17.135017    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:17.358947 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:17.358883    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:17.613624 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:17.613589    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:18.167361 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.167336    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 19:07:18.193351 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.193317    4924 generic.go:356] "Generic (PLEG): container finished" podID="ccd81788-05eb-4696-938f-d6dc6d3ab2ce" containerID="a20ef174b15c0d5c8dd4912907e77576e9907973a6773c5a6bbb2f624b806b7b" exitCode=0
Oct 30 19:07:18.193549 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.193360    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw" event={"ID":"ccd81788-05eb-4696-938f-d6dc6d3ab2ce","Type":"ContainerDied","Data":"a20ef174b15c0d5c8dd4912907e77576e9907973a6773c5a6bbb2f624b806b7b"}
Oct 30 19:07:18.193549 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.193425    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw" event={"ID":"ccd81788-05eb-4696-938f-d6dc6d3ab2ce","Type":"ContainerDied","Data":"db2a7da134ec1e27348f23a484be62990061c4df722ea179040640bb0f8da161"}
Oct 30 19:07:18.193549 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.193436    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"
Oct 30 19:07:18.193549 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.193456    4924 scope.go:122] "RemoveContainer" containerID="a20ef174b15c0d5c8dd4912907e77576e9907973a6773c5a6bbb2f624b806b7b"
Oct 30 19:07:18.205009 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.204982    4924 scope.go:122] "RemoveContainer" containerID="a20ef174b15c0d5c8dd4912907e77576e9907973a6773c5a6bbb2f624b806b7b"
Oct 30 19:07:18.205332 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:07:18.205300    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a20ef174b15c0d5c8dd4912907e77576e9907973a6773c5a6bbb2f624b806b7b\": container with ID starting with a20ef174b15c0d5c8dd4912907e77576e9907973a6773c5a6bbb2f624b806b7b not found: ID does not exist" containerID="a20ef174b15c0d5c8dd4912907e77576e9907973a6773c5a6bbb2f624b806b7b"
Oct 30 19:07:18.205405 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.205342    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a20ef174b15c0d5c8dd4912907e77576e9907973a6773c5a6bbb2f624b806b7b"} err="failed to get container status \"a20ef174b15c0d5c8dd4912907e77576e9907973a6773c5a6bbb2f624b806b7b\": rpc error: code = NotFound desc = could not find container \"a20ef174b15c0d5c8dd4912907e77576e9907973a6773c5a6bbb2f624b806b7b\": container with ID starting with a20ef174b15c0d5c8dd4912907e77576e9907973a6773c5a6bbb2f624b806b7b not found: ID does not exist"
Oct 30 19:07:18.300173 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.300142    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-token\" (UniqueName: \"kubernetes.io/projected/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-token\") pod \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") "
Oct 30 19:07:18.300341 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.300250    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-workload-certs\") pod \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") "
Oct 30 19:07:18.300341 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.300323    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-kube-api-access-l994m\" (UniqueName: \"kubernetes.io/projected/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-kube-api-access-l994m\") pod \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") "
Oct 30 19:07:18.300434 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.300364    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istiod-ca-cert\") pod \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") "
Oct 30 19:07:18.300470 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.300448    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-podinfo\") pod \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") "
Oct 30 19:07:18.300542 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.300521    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-credential-socket\") pod \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") "
Oct 30 19:07:18.300607 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.300579    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-workload-certs" pod "ccd81788-05eb-4696-938f-d6dc6d3ab2ce" (UID: "ccd81788-05eb-4696-938f-d6dc6d3ab2ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:18.300685 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.300604    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-data\") pod \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") "
Oct 30 19:07:18.300753 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.300731    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-envoy\") pod \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") "
Oct 30 19:07:18.300853 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.300832    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-data" pod "ccd81788-05eb-4696-938f-d6dc6d3ab2ce" (UID: "ccd81788-05eb-4696-938f-d6dc6d3ab2ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:18.300938 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.300847    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-credential-socket" pod "ccd81788-05eb-4696-938f-d6dc6d3ab2ce" (UID: "ccd81788-05eb-4696-938f-d6dc6d3ab2ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:18.300938 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.300870    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istiod-ca-cert" pod "ccd81788-05eb-4696-938f-d6dc6d3ab2ce" (UID: "ccd81788-05eb-4696-938f-d6dc6d3ab2ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:18.301088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.301039    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-workload-socket\") pod \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\" (UID: \"ccd81788-05eb-4696-938f-d6dc6d3ab2ce\") "
Oct 30 19:07:18.301313 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.301290    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-workload-socket" pod "ccd81788-05eb-4696-938f-d6dc6d3ab2ce" (UID: "ccd81788-05eb-4696-938f-d6dc6d3ab2ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:18.301975 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.301947    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:18.302112 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.301980    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:18.302112 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.301997    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:18.302112 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.302011    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:18.302112 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.302025    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:18.302702 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.302677    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-token" pod "ccd81788-05eb-4696-938f-d6dc6d3ab2ce" (UID: "ccd81788-05eb-4696-938f-d6dc6d3ab2ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:18.302931 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.302912    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-envoy" pod "ccd81788-05eb-4696-938f-d6dc6d3ab2ce" (UID: "ccd81788-05eb-4696-938f-d6dc6d3ab2ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:18.303107 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.303083    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-podinfo" pod "ccd81788-05eb-4696-938f-d6dc6d3ab2ce" (UID: "ccd81788-05eb-4696-938f-d6dc6d3ab2ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:07:18.303186 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.303138    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-kube-api-access-l994m" pod "ccd81788-05eb-4696-938f-d6dc6d3ab2ce" (UID: "ccd81788-05eb-4696-938f-d6dc6d3ab2ce"). InnerVolumeSpecName "kube-api-access-l994m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:18.404490 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.404411    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:18.404490 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.404440    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:18.404490 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.404450    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l994m\" (UniqueName: \"kubernetes.io/projected/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-kube-api-access-l994m\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:18.404490 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.404461    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd81788-05eb-4696-938f-d6dc6d3ab2ce-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:18.614182 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.614143    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"]
Oct 30 19:07:18.616311 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.616279    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-xswrw"]
Oct 30 19:07:18.838375 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:18.838289    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd81788-05eb-4696-938f-d6dc6d3ab2ce" path="/var/lib/kubelet/pods/ccd81788-05eb-4696-938f-d6dc6d3ab2ce/volumes"
Oct 30 19:07:22.774384 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:22.774286    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:25.784963 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:25.784925    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:26.813124 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:26.813089    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:27.960260 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:27.960222    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2299c938-vn9l7"]
Oct 30 19:07:27.962387 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:27.962357    4924 cpu_manager.go:414] "containerMap: removing container" podUID="77664ce4-7a08-4b33-bccf-2e0755f13be3" containerName="istio-proxy"
Oct 30 19:07:27.962561 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:27.962547    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77664ce4-7a08-4b33-bccf-2e0755f13be3" containerName="istio-proxy"
Oct 30 19:07:27.962733 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:27.962721    4924 cpu_manager.go:414] "containerMap: removing container" podUID="ccd81788-05eb-4696-938f-d6dc6d3ab2ce" containerName="istio-proxy"
Oct 30 19:07:27.962811 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:27.962800    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd81788-05eb-4696-938f-d6dc6d3ab2ce" containerName="istio-proxy"
Oct 30 19:07:27.962933 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:27.962924    4924 cpu_manager.go:414] "containerMap: removing container" podUID="77664ce4-7a08-4b33-bccf-2e0755f13be3" containerName="istio-proxy"
Oct 30 19:07:27.963001 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:27.962992    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77664ce4-7a08-4b33-bccf-2e0755f13be3" containerName="istio-proxy"
Oct 30 19:07:27.963442 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:27.963419    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="77664ce4-7a08-4b33-bccf-2e0755f13be3" containerName="istio-proxy"
Oct 30 19:07:27.963601 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:27.963588    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="61957487-4e2c-47d7-9977-1d34f3c2534f" containerName="istio-proxy"
Oct 30 19:07:27.963698 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:27.963688    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd81788-05eb-4696-938f-d6dc6d3ab2ce" containerName="istio-proxy"
Oct 30 19:07:27.969659 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:27.969640    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2299c938-vn9l7"
Oct 30 19:07:28.001920 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:28.001896    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"]
Oct 30 19:07:28.002120 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:28.002087    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc" podUID="3d89616c-b699-44df-983a-d58cad14b601" containerName="istio-proxy" containerID="cri-o://69adc337b05af0ac65343a1ac235e49fc34c7a31d79624240d838473650d589a" gracePeriod=30
Oct 30 19:07:28.015078 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:28.015042    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2299c938-vn9l7"]
Oct 30 19:07:28.040481 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:28.040454    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:07:28.040644 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:28.040631    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:07:28.040712 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:28.040652    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:07:28.044013 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:28.043995    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:07:28.067621 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:28.067589    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3175550a-63b4-4493-a54c-1748e7626d88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2299c938-vn9l7\" (UID: \"3175550a-63b4-4493-a54c-1748e7626d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2299c938-vn9l7"
Oct 30 19:07:28.067861 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:28.067844    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22ppk\" (UniqueName: \"kubernetes.io/projected/3175550a-63b4-4493-a54c-1748e7626d88-kube-api-access-22ppk\") pod \"maas-api-revoke-sub-2299c938-vn9l7\" (UID: \"3175550a-63b4-4493-a54c-1748e7626d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2299c938-vn9l7"
Oct 30 19:07:28.067995 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:28.067973    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3175550a-63b4-4493-a54c-1748e7626d88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2299c938-vn9l7\" (UID: \"3175550a-63b4-4493-a54c-1748e7626d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2299c938-vn9l7"
Oct 30 19:07:28.169499 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:28.169465    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22ppk\" (UniqueName: \"kubernetes.io/projected/3175550a-63b4-4493-a54c-1748e7626d88-kube-api-access-22ppk\") pod \"maas-api-revoke-sub-2299c938-vn9l7\" (UID: \"3175550a-63b4-4493-a54c-1748e7626d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2299c938-vn9l7"
Oct 30 19:07:28.169675 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:28.169562    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3175550a-63b4-4493-a54c-1748e7626d88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2299c938-vn9l7\" (UID: \"3175550a-63b4-4493-a54c-1748e7626d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2299c938-vn9l7"
Oct 30 19:07:28.169719 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:28.169680    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3175550a-63b4-4493-a54c-1748e7626d88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2299c938-vn9l7\" (UID: \"3175550a-63b4-4493-a54c-1748e7626d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2299c938-vn9l7"
Oct 30 19:07:28.170339 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:28.170316    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3175550a-63b4-4493-a54c-1748e7626d88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2299c938-vn9l7\" (UID: \"3175550a-63b4-4493-a54c-1748e7626d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2299c938-vn9l7"
Oct 30 19:07:28.172014 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:28.171988    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3175550a-63b4-4493-a54c-1748e7626d88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2299c938-vn9l7\" (UID: \"3175550a-63b4-4493-a54c-1748e7626d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2299c938-vn9l7"
Oct 30 19:07:28.211277 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:28.211186    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:28.226528 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:28.226494    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22ppk\" (UniqueName: \"kubernetes.io/projected/3175550a-63b4-4493-a54c-1748e7626d88-kube-api-access-22ppk\") pod \"maas-api-revoke-sub-2299c938-vn9l7\" (UID: \"3175550a-63b4-4493-a54c-1748e7626d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2299c938-vn9l7"
Oct 30 19:07:28.284515 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:28.284473    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2299c938-vn9l7"
Oct 30 19:07:28.712347 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:28.712316    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2299c938-vn9l7"]
Oct 30 19:07:28.712669 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:07:28.712645    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3175550a_63b4_4493_a54c_1748e7626d88.slice/crio-aa625e1649290a9d8340a15b346a7f1a11ae0229dd4a26e20a62a20a36ccee00 WatchSource:0}: Error finding container aa625e1649290a9d8340a15b346a7f1a11ae0229dd4a26e20a62a20a36ccee00: Status 404 returned error can't find the container with id aa625e1649290a9d8340a15b346a7f1a11ae0229dd4a26e20a62a20a36ccee00
Oct 30 19:07:29.287403 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:29.287314    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2299c938-vn9l7" event={"ID":"3175550a-63b4-4493-a54c-1748e7626d88","Type":"ContainerStarted","Data":"dd568467e6e9f1d8d331fabb3622b7df31f7a6d070f6d00317a15f59dfb480cc"}
Oct 30 19:07:29.287403 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:29.287375    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2299c938-vn9l7" event={"ID":"3175550a-63b4-4493-a54c-1748e7626d88","Type":"ContainerStarted","Data":"aa625e1649290a9d8340a15b346a7f1a11ae0229dd4a26e20a62a20a36ccee00"}
Oct 30 19:07:29.396878 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:29.396827    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2299c938-vn9l7" podStartSLOduration=2.396813919 podStartE2EDuration="2.396813919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:29.393810192 +0000 UTC m=+1850.670860637" watchObservedRunningTime="2026-10-30 19:07:29.396813919 +0000 UTC m=+1850.673864363"
Oct 30 19:07:30.297181 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:30.297142    4924 generic.go:356] "Generic (PLEG): container finished" podID="3175550a-63b4-4493-a54c-1748e7626d88" containerID="dd568467e6e9f1d8d331fabb3622b7df31f7a6d070f6d00317a15f59dfb480cc" exitCode=0
Oct 30 19:07:30.297613 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:30.297230    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2299c938-vn9l7" event={"ID":"3175550a-63b4-4493-a54c-1748e7626d88","Type":"ContainerDied","Data":"dd568467e6e9f1d8d331fabb3622b7df31f7a6d070f6d00317a15f59dfb480cc"}
Oct 30 19:07:31.468794 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:31.468767    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2299c938-vn9l7"
Oct 30 19:07:31.527249 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:31.526999    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3175550a-63b4-4493-a54c-1748e7626d88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3175550a-63b4-4493-a54c-1748e7626d88-maas-api-xks-ca\") pod \"3175550a-63b4-4493-a54c-1748e7626d88\" (UID: \"3175550a-63b4-4493-a54c-1748e7626d88\") "
Oct 30 19:07:31.527445 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:31.527357    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3175550a-63b4-4493-a54c-1748e7626d88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3175550a-63b4-4493-a54c-1748e7626d88-maas-api-openshift-ca\") pod \"3175550a-63b4-4493-a54c-1748e7626d88\" (UID: \"3175550a-63b4-4493-a54c-1748e7626d88\") "
Oct 30 19:07:31.531028 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:31.527555    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3175550a-63b4-4493-a54c-1748e7626d88-kube-api-access-22ppk\" (UniqueName: \"kubernetes.io/projected/3175550a-63b4-4493-a54c-1748e7626d88-kube-api-access-22ppk\") pod \"3175550a-63b4-4493-a54c-1748e7626d88\" (UID: \"3175550a-63b4-4493-a54c-1748e7626d88\") "
Oct 30 19:07:31.538255 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:31.538215    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3175550a-63b4-4493-a54c-1748e7626d88-kube-api-access-22ppk" pod "3175550a-63b4-4493-a54c-1748e7626d88" (UID: "3175550a-63b4-4493-a54c-1748e7626d88"). InnerVolumeSpecName "kube-api-access-22ppk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:31.538966 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:31.538927    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3175550a-63b4-4493-a54c-1748e7626d88-maas-api-openshift-ca" pod "3175550a-63b4-4493-a54c-1748e7626d88" (UID: "3175550a-63b4-4493-a54c-1748e7626d88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:31.539894 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:31.539866    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3175550a-63b4-4493-a54c-1748e7626d88-maas-api-xks-ca" pod "3175550a-63b4-4493-a54c-1748e7626d88" (UID: "3175550a-63b4-4493-a54c-1748e7626d88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:07:31.631136 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:31.631101    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22ppk\" (UniqueName: \"kubernetes.io/projected/3175550a-63b4-4493-a54c-1748e7626d88-kube-api-access-22ppk\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:31.631136 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:31.631140    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3175550a-63b4-4493-a54c-1748e7626d88-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:31.631332 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:31.631155    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3175550a-63b4-4493-a54c-1748e7626d88-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:32.314616 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:32.314584    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2299c938-vn9l7"
Oct 30 19:07:32.314821 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:32.314615    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2299c938-vn9l7" event={"ID":"3175550a-63b4-4493-a54c-1748e7626d88","Type":"ContainerDied","Data":"aa625e1649290a9d8340a15b346a7f1a11ae0229dd4a26e20a62a20a36ccee00"}
Oct 30 19:07:32.314821 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:32.314674    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa625e1649290a9d8340a15b346a7f1a11ae0229dd4a26e20a62a20a36ccee00"
Oct 30 19:07:32.409091 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:32.404648    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:33.279883 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.279818    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 19:07:33.324966 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.324928    4924 generic.go:356] "Generic (PLEG): container finished" podID="3d89616c-b699-44df-983a-d58cad14b601" containerID="69adc337b05af0ac65343a1ac235e49fc34c7a31d79624240d838473650d589a" exitCode=0
Oct 30 19:07:33.325182 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.324975    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"
Oct 30 19:07:33.325182 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.325012    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc" event={"ID":"3d89616c-b699-44df-983a-d58cad14b601","Type":"ContainerDied","Data":"69adc337b05af0ac65343a1ac235e49fc34c7a31d79624240d838473650d589a"}
Oct 30 19:07:33.325182 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.325083    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc" event={"ID":"3d89616c-b699-44df-983a-d58cad14b601","Type":"ContainerDied","Data":"b3ab0d9e08f7374c8c3e8d8261b532d55feeab110f50e6ae125e43a95a9dd865"}
Oct 30 19:07:33.325182 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.325108    4924 scope.go:122] "RemoveContainer" containerID="69adc337b05af0ac65343a1ac235e49fc34c7a31d79624240d838473650d589a"
Oct 30 19:07:33.337511 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.337489    4924 scope.go:122] "RemoveContainer" containerID="69adc337b05af0ac65343a1ac235e49fc34c7a31d79624240d838473650d589a"
Oct 30 19:07:33.337806 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:07:33.337785    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69adc337b05af0ac65343a1ac235e49fc34c7a31d79624240d838473650d589a\": container with ID starting with 69adc337b05af0ac65343a1ac235e49fc34c7a31d79624240d838473650d589a not found: ID does not exist" containerID="69adc337b05af0ac65343a1ac235e49fc34c7a31d79624240d838473650d589a"
Oct 30 19:07:33.337902 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.337814    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69adc337b05af0ac65343a1ac235e49fc34c7a31d79624240d838473650d589a"} err="failed to get container status \"69adc337b05af0ac65343a1ac235e49fc34c7a31d79624240d838473650d589a\": rpc error: code = NotFound desc = could not find container \"69adc337b05af0ac65343a1ac235e49fc34c7a31d79624240d838473650d589a\": container with ID starting with 69adc337b05af0ac65343a1ac235e49fc34c7a31d79624240d838473650d589a not found: ID does not exist"
Oct 30 19:07:33.368089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.366323    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-istio-data\") pod \"3d89616c-b699-44df-983a-d58cad14b601\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") "
Oct 30 19:07:33.368089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.366668    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-istio-data" pod "3d89616c-b699-44df-983a-d58cad14b601" (UID: "3d89616c-b699-44df-983a-d58cad14b601"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:33.368089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.366878    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d89616c-b699-44df-983a-d58cad14b601-kube-api-access-jttf5\" (UniqueName: \"kubernetes.io/projected/3d89616c-b699-44df-983a-d58cad14b601-kube-api-access-jttf5\") pod \"3d89616c-b699-44df-983a-d58cad14b601\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") "
Oct 30 19:07:33.368089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.366972    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-credential-socket\") pod \"3d89616c-b699-44df-983a-d58cad14b601\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") "
Oct 30 19:07:33.368089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.367016    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-workload-socket\") pod \"3d89616c-b699-44df-983a-d58cad14b601\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") "
Oct 30 19:07:33.368089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.367046    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d89616c-b699-44df-983a-d58cad14b601-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d89616c-b699-44df-983a-d58cad14b601-istiod-ca-cert\") pod \"3d89616c-b699-44df-983a-d58cad14b601\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") "
Oct 30 19:07:33.368089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.367152    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d89616c-b699-44df-983a-d58cad14b601-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d89616c-b699-44df-983a-d58cad14b601-istio-podinfo\") pod \"3d89616c-b699-44df-983a-d58cad14b601\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") "
Oct 30 19:07:33.368089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.367349    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-workload-socket" pod "3d89616c-b699-44df-983a-d58cad14b601" (UID: "3d89616c-b699-44df-983a-d58cad14b601"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:33.368089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.367512    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d89616c-b699-44df-983a-d58cad14b601-istiod-ca-cert" pod "3d89616c-b699-44df-983a-d58cad14b601" (UID: "3d89616c-b699-44df-983a-d58cad14b601"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:07:33.368089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.367580    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-workload-certs\") pod \"3d89616c-b699-44df-983a-d58cad14b601\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") "
Oct 30 19:07:33.368089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.367639    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d89616c-b699-44df-983a-d58cad14b601-istio-token\" (UniqueName: \"kubernetes.io/projected/3d89616c-b699-44df-983a-d58cad14b601-istio-token\") pod \"3d89616c-b699-44df-983a-d58cad14b601\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") "
Oct 30 19:07:33.368089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.367730    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-istio-envoy\") pod \"3d89616c-b699-44df-983a-d58cad14b601\" (UID: \"3d89616c-b699-44df-983a-d58cad14b601\") "
Oct 30 19:07:33.368089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.367951    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-workload-certs" pod "3d89616c-b699-44df-983a-d58cad14b601" (UID: "3d89616c-b699-44df-983a-d58cad14b601"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:33.368089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.367985    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-credential-socket" pod "3d89616c-b699-44df-983a-d58cad14b601" (UID: "3d89616c-b699-44df-983a-d58cad14b601"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:33.368974 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.368680    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:33.368974 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.368706    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:33.368974 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.368720    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:33.368974 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.368734    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d89616c-b699-44df-983a-d58cad14b601-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:33.368974 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.368747    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:33.375084 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.374437    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-istio-envoy" pod "3d89616c-b699-44df-983a-d58cad14b601" (UID: "3d89616c-b699-44df-983a-d58cad14b601"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:07:33.375084 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.374566    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d89616c-b699-44df-983a-d58cad14b601-istio-podinfo" pod "3d89616c-b699-44df-983a-d58cad14b601" (UID: "3d89616c-b699-44df-983a-d58cad14b601"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:07:33.375084 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.374684    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d89616c-b699-44df-983a-d58cad14b601-kube-api-access-jttf5" pod "3d89616c-b699-44df-983a-d58cad14b601" (UID: "3d89616c-b699-44df-983a-d58cad14b601"). InnerVolumeSpecName "kube-api-access-jttf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:33.399295 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.397877    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d89616c-b699-44df-983a-d58cad14b601-istio-token" pod "3d89616c-b699-44df-983a-d58cad14b601" (UID: "3d89616c-b699-44df-983a-d58cad14b601"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:07:33.484087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.479533    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d89616c-b699-44df-983a-d58cad14b601-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:33.484087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.479575    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jttf5\" (UniqueName: \"kubernetes.io/projected/3d89616c-b699-44df-983a-d58cad14b601-kube-api-access-jttf5\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:33.484087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.479592    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d89616c-b699-44df-983a-d58cad14b601-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:33.484087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.479608    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d89616c-b699-44df-983a-d58cad14b601-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:07:33.634143 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.633515    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:33.783314 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.783279    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"]
Oct 30 19:07:33.792574 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:33.792536    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-lxtcc"]
Oct 30 19:07:34.836418 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:34.836387    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d89616c-b699-44df-983a-d58cad14b601" path="/var/lib/kubelet/pods/3d89616c-b699-44df-983a-d58cad14b601/volumes"
Oct 30 19:07:38.676989 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:38.676954    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:38.899153 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:38.898955    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:39.111323 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:39.111285    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:39.559558 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:39.559478    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:39.959772 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:39.959738    4924 scope.go:122] "RemoveContainer" containerID="e6439e1459a4a5efbb437fd1384f576c3d719f8508898c09d90494f2a12d48a7"
Oct 30 19:07:39.973497 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:39.973374    4924 scope.go:122] "RemoveContainer" containerID="e11f0a95a4afbca41bdf6d4edf3d1a90c0f59ca1baac2b92189d16353936c009"
Oct 30 19:07:39.983672 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:39.983653    4924 scope.go:122] "RemoveContainer" containerID="627a98df6325d22453182de899ad5d8976c021f15ec508aec78c48e9400198d8"
Oct 30 19:07:39.993563 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:39.993539    4924 scope.go:122] "RemoveContainer" containerID="fa5e99019a7cd4614ba94f2bafa18dd25f1eeed29ef59a542f17bf4e833fe9b0"
Oct 30 19:07:40.002446 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:40.002427    4924 scope.go:122] "RemoveContainer" containerID="952a4c0f360c3fade3052485d828cff0caabb8a20c3461a4d269f4584334d42b"
Oct 30 19:07:43.889422 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:43.889380    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:45.713800 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:07:45.713765    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:07:46.083670 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:46.083593    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:46.562922 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:46.562891    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:47.340378 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:47.339773    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:48.019967 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:48.019938    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:49.314582 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:49.314530    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:51.080082 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:51.080036    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:52.192655 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:52.192618    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:52.441604 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:52.441571    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:52.726886 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:52.726841    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:53.376605 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:53.376574    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:53.716410 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:53.716332    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:54.933056 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:54.932940    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:55.697771 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:55.697722    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:56.030058 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:56.029948    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:58.066773 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:58.066739    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:07:58.369201 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:07:58.369168    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:00.824136 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:00.824089    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:01.127873 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:01.127838    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:01.586886 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:01.586780    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:01.830056 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:01.830016    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:02.050973 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:02.050933    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz"]
Oct 30 19:08:02.052049 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:02.052031    4924 cpu_manager.go:414] "containerMap: removing container" podUID="3d89616c-b699-44df-983a-d58cad14b601" containerName="istio-proxy"
Oct 30 19:08:02.052128 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:02.052052    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d89616c-b699-44df-983a-d58cad14b601" containerName="istio-proxy"
Oct 30 19:08:02.052128 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:02.052085    4924 cpu_manager.go:414] "containerMap: removing container" podUID="3175550a-63b4-4493-a54c-1748e7626d88" containerName="revoke-keys"
Oct 30 19:08:02.052128 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:02.052093    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3175550a-63b4-4493-a54c-1748e7626d88" containerName="revoke-keys"
Oct 30 19:08:02.052269 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:02.052255    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="77664ce4-7a08-4b33-bccf-2e0755f13be3" containerName="istio-proxy"
Oct 30 19:08:02.052310 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:02.052275    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d89616c-b699-44df-983a-d58cad14b601" containerName="istio-proxy"
Oct 30 19:08:02.052310 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:02.052289    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="3175550a-63b4-4493-a54c-1748e7626d88" containerName="revoke-keys"
Oct 30 19:08:02.056579 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:02.056548    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz"
Oct 30 19:08:02.106881 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:02.106673    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kvmh\" (UniqueName: \"kubernetes.io/projected/363c9626-84ae-4605-95a3-7c000aebe1af-kube-api-access-8kvmh\") pod \"payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz\" (UID: \"363c9626-84ae-4605-95a3-7c000aebe1af\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz"
Oct 30 19:08:02.106881 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:02.106832    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363c9626-84ae-4605-95a3-7c000aebe1af-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz\" (UID: \"363c9626-84ae-4605-95a3-7c000aebe1af\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz"
Oct 30 19:08:02.171115 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:02.163353    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-cc079d-dockercfg-c928t\""
Oct 30 19:08:02.199151 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:02.169646    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-cc079d\""
Oct 30 19:08:02.254090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:02.248718    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kvmh\" (UniqueName: \"kubernetes.io/projected/363c9626-84ae-4605-95a3-7c000aebe1af-kube-api-access-8kvmh\") pod \"payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz\" (UID: \"363c9626-84ae-4605-95a3-7c000aebe1af\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz"
Oct 30 19:08:02.254090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:02.248884    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363c9626-84ae-4605-95a3-7c000aebe1af-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz\" (UID: \"363c9626-84ae-4605-95a3-7c000aebe1af\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz"
Oct 30 19:08:02.254090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:02.251418    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363c9626-84ae-4605-95a3-7c000aebe1af-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz\" (UID: \"363c9626-84ae-4605-95a3-7c000aebe1af\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz"
Oct 30 19:08:02.334193 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:02.332030    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz"]
Oct 30 19:08:02.334193 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:02.332273    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kvmh\" (UniqueName: \"kubernetes.io/projected/363c9626-84ae-4605-95a3-7c000aebe1af-kube-api-access-8kvmh\") pod \"payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz\" (UID: \"363c9626-84ae-4605-95a3-7c000aebe1af\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz"
Oct 30 19:08:02.382132 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:02.372913    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz"
Oct 30 19:08:02.793862 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:02.793829    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz"]
Oct 30 19:08:02.794043 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:08:02.794018    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363c9626_84ae_4605_95a3_7c000aebe1af.slice/crio-b3c0dc2832561f9d58ec27427480e5ecb8f39e846bd131ff5cfcd067451c0ff4 WatchSource:0}: Error finding container b3c0dc2832561f9d58ec27427480e5ecb8f39e846bd131ff5cfcd067451c0ff4: Status 404 returned error can't find the container with id b3c0dc2832561f9d58ec27427480e5ecb8f39e846bd131ff5cfcd067451c0ff4
Oct 30 19:08:02.942444 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:02.942411    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz" event={"ID":"363c9626-84ae-4605-95a3-7c000aebe1af","Type":"ContainerStarted","Data":"e48476a0a92c7e93145c3c4d9f07ddede805cefcfedd80c9c7436978c4a2fd31"}
Oct 30 19:08:02.942857 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:02.942467    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz" event={"ID":"363c9626-84ae-4605-95a3-7c000aebe1af","Type":"ContainerStarted","Data":"b3c0dc2832561f9d58ec27427480e5ecb8f39e846bd131ff5cfcd067451c0ff4"}
Oct 30 19:08:02.942857 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:02.942496    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz"
Oct 30 19:08:03.060688 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:03.060583    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz" podStartSLOduration=1.060567196 podStartE2EDuration="1.060567196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:03.058700689 +0000 UTC m=+1884.335751134" watchObservedRunningTime="2026-10-30 19:08:03.060567196 +0000 UTC m=+1884.337617645"
Oct 30 19:08:06.943038 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:06.942993    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:11.798173 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:11.798137    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:13.951888 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:13.951856    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz"
Oct 30 19:08:15.059326 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:15.059290    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8105ddbb-r2sfr"]
Oct 30 19:08:15.063482 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:15.063457    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8105ddbb-r2sfr"]
Oct 30 19:08:15.714273 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:08:15.714235    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:08:16.644230 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:16.644190    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:16.837488 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:16.837455    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b5ba138-0529-4505-8ede-215d7e3c1b7a" path="/var/lib/kubelet/pods/9b5ba138-0529-4505-8ede-215d7e3c1b7a/volumes"
Oct 30 19:08:22.130938 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:22.130905    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:22.847230 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:22.847200    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:27.628830 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:27.628794    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:27.871923 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:27.871886    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:28.645166 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:28.645134    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:32.991460 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:32.991425    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:36.921280 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:36.921248    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:37.970193 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:37.970144    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:38.179115 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:38.179027    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:38.989372 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:38.989341    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:40.066378 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:40.066347    4924 scope.go:122] "RemoveContainer" containerID="7d0684ac3f957a1089b3acccc15133ae523469cb641bfc1dfaab2c3d3506adaf"
Oct 30 19:08:40.081185 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:40.081161    4924 scope.go:122] "RemoveContainer" containerID="c409b0878e573af81af78dd42cb916eb28793520dff805bbf6cfb8981132398c"
Oct 30 19:08:42.044800 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:42.044746    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:42.274249 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:42.274193    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:43.118373 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:43.118335    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:45.164219 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:45.164183    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:45.449917 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:45.449833    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:45.912777 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:08:45.912723    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:08:48.620293 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:48.620258    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:53.822438 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:53.822356    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:08:58.638807 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:08:58.638769    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:01.052532 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:01.052497    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8040-txg9n"]
Oct 30 19:09:01.099059 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:01.099025    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8040-txg9n"]
Oct 30 19:09:02.840576 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:02.840527    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="001c4e0f-57d2-4624-8c64-2f5afaaabeba" path="/var/lib/kubelet/pods/001c4e0f-57d2-4624-8c64-2f5afaaabeba/volumes"
Oct 30 19:09:03.457301 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:03.457264    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:08.382668 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:08.382604    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:08.877184 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:08.877154    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:09.067254 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:09.067225    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:13.175120 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:13.175048    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:13.377965 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:13.377930    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:13.576728 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:13.576641    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"]
Oct 30 19:09:13.577200 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:13.577043    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx" podUID="53b2edae-97f1-44e9-9580-29d605357c8f" containerName="istio-proxy" containerID="cri-o://5ba0865c313b090504e1e17e13cfe38c45726c245351ba777b4d11feb53c5cb8" gracePeriod=30
Oct 30 19:09:13.629926 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:13.629895    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"]
Oct 30 19:09:13.630136 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:13.630110    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p" podUID="2eaee46c-23ef-410a-a5b6-450eba9711cc" containerName="istio-proxy" containerID="cri-o://b82e787d0d3b24eff24c9b84268fd81e2d3db47b25dbffb04823654bdf6863ca" gracePeriod=30
Oct 30 19:09:13.779361 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:13.779327    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"]
Oct 30 19:09:13.779601 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:13.779551    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2" podUID="34dfd0fe-b500-4df5-b191-6085a5d8dde5" containerName="istio-proxy" containerID="cri-o://6a7aa424319bce6c1197a1f465f648982b85daafa1d99bfbb22483f3a46178df" gracePeriod=30
Oct 30 19:09:13.783525 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:13.783499    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"]
Oct 30 19:09:13.783674 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:13.783653    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk" podUID="7d743094-1851-4c3a-9866-ecb1062c69b6" containerName="istio-proxy" containerID="cri-o://d898f5123e81a57cd207d07ece648447a1cb0b5e632a8f96ce1333f52d89d449" gracePeriod=30
Oct 30 19:09:13.913391 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:13.913358    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"]
Oct 30 19:09:13.914134 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:13.913589    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw" podUID="922dfc11-a4ba-4d50-a50f-b5155d277792" containerName="istio-proxy" containerID="cri-o://51132c3f0c9a06d65bd4696f29e0ad1db4388b8946da5d03eb28807462cef97c" gracePeriod=30
Oct 30 19:09:15.713023 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:09:15.712988    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:09:17.333054 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:17.333020    4924 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:09:17.333476 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:17.333113    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p" podUID="2eaee46c-23ef-410a-a5b6-450eba9711cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:09:18.202278 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:18.202239    4924 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:09:18.202454 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:18.202294    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk" podUID="7d743094-1851-4c3a-9866-ecb1062c69b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:09:18.344864 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:18.344821    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:18.623370 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:18.623342    4924 patch_prober.go:29] interesting pod/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:09:18.623519 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:18.623390    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2" podUID="34dfd0fe-b500-4df5-b191-6085a5d8dde5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:09:19.008729 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.008696    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:09:19.036153 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.036126    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 19:09:19.097250 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.097220    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 19:09:19.117220 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.116550    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-credential-socket\") pod \"53b2edae-97f1-44e9-9580-29d605357c8f\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") "
Oct 30 19:09:19.117220 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.116636    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-credential-socket\") pod \"2eaee46c-23ef-410a-a5b6-450eba9711cc\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") "
Oct 30 19:09:19.117220 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.116702    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-workload-socket\") pod \"2eaee46c-23ef-410a-a5b6-450eba9711cc\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") "
Oct 30 19:09:19.117220 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.116751    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-workload-certs\") pod \"2eaee46c-23ef-410a-a5b6-450eba9711cc\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") "
Oct 30 19:09:19.117220 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.116783    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-workload-socket\") pod \"53b2edae-97f1-44e9-9580-29d605357c8f\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") "
Oct 30 19:09:19.117220 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.116859    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-envoy\") pod \"2eaee46c-23ef-410a-a5b6-450eba9711cc\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") "
Oct 30 19:09:19.117220 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.116899    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53b2edae-97f1-44e9-9580-29d605357c8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53b2edae-97f1-44e9-9580-29d605357c8f-istiod-ca-cert\") pod \"53b2edae-97f1-44e9-9580-29d605357c8f\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") "
Oct 30 19:09:19.117220 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.117030    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-data\") pod \"2eaee46c-23ef-410a-a5b6-450eba9711cc\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") "
Oct 30 19:09:19.117220 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.117151    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-istio-data\") pod \"53b2edae-97f1-44e9-9580-29d605357c8f\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") "
Oct 30 19:09:19.117758 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.117259    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-token\" (UniqueName: \"kubernetes.io/projected/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-token\") pod \"2eaee46c-23ef-410a-a5b6-450eba9711cc\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") "
Oct 30 19:09:19.117758 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.117288    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-istio-envoy\") pod \"53b2edae-97f1-44e9-9580-29d605357c8f\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") "
Oct 30 19:09:19.117758 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.117349    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53b2edae-97f1-44e9-9580-29d605357c8f-kube-api-access-5vwbp\" (UniqueName: \"kubernetes.io/projected/53b2edae-97f1-44e9-9580-29d605357c8f-kube-api-access-5vwbp\") pod \"53b2edae-97f1-44e9-9580-29d605357c8f\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") "
Oct 30 19:09:19.117758 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.117386    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53b2edae-97f1-44e9-9580-29d605357c8f-istio-token\" (UniqueName: \"kubernetes.io/projected/53b2edae-97f1-44e9-9580-29d605357c8f-istio-token\") pod \"53b2edae-97f1-44e9-9580-29d605357c8f\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") "
Oct 30 19:09:19.117758 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.117443    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-podinfo\") pod \"2eaee46c-23ef-410a-a5b6-450eba9711cc\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") "
Oct 30 19:09:19.117758 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.117480    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eaee46c-23ef-410a-a5b6-450eba9711cc-kube-api-access-zs7z9\" (UniqueName: \"kubernetes.io/projected/2eaee46c-23ef-410a-a5b6-450eba9711cc-kube-api-access-zs7z9\") pod \"2eaee46c-23ef-410a-a5b6-450eba9711cc\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") "
Oct 30 19:09:19.117758 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.117541    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eaee46c-23ef-410a-a5b6-450eba9711cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eaee46c-23ef-410a-a5b6-450eba9711cc-istiod-ca-cert\") pod \"2eaee46c-23ef-410a-a5b6-450eba9711cc\" (UID: \"2eaee46c-23ef-410a-a5b6-450eba9711cc\") "
Oct 30 19:09:19.117758 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.117592    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-workload-certs\") pod \"53b2edae-97f1-44e9-9580-29d605357c8f\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") "
Oct 30 19:09:19.117758 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.117644    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53b2edae-97f1-44e9-9580-29d605357c8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53b2edae-97f1-44e9-9580-29d605357c8f-istio-podinfo\") pod \"53b2edae-97f1-44e9-9580-29d605357c8f\" (UID: \"53b2edae-97f1-44e9-9580-29d605357c8f\") "
Oct 30 19:09:19.118332 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.118011    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-istio-data" pod "53b2edae-97f1-44e9-9580-29d605357c8f" (UID: "53b2edae-97f1-44e9-9580-29d605357c8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.118332 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.118229    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-credential-socket" pod "53b2edae-97f1-44e9-9580-29d605357c8f" (UID: "53b2edae-97f1-44e9-9580-29d605357c8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.118444 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.118418    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-credential-socket" pod "2eaee46c-23ef-410a-a5b6-450eba9711cc" (UID: "2eaee46c-23ef-410a-a5b6-450eba9711cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.118655 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.118627    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-workload-socket" pod "2eaee46c-23ef-410a-a5b6-450eba9711cc" (UID: "2eaee46c-23ef-410a-a5b6-450eba9711cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.118837 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.118809    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-workload-certs" pod "2eaee46c-23ef-410a-a5b6-450eba9711cc" (UID: "2eaee46c-23ef-410a-a5b6-450eba9711cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.119028 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.119008    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-workload-socket" pod "53b2edae-97f1-44e9-9580-29d605357c8f" (UID: "53b2edae-97f1-44e9-9580-29d605357c8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.120013 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.119989    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.120163 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.120019    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.120163 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.120036    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.120163 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.120050    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.120163 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.120080    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.120163 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.120096    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.121113 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.120782    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-data" pod "2eaee46c-23ef-410a-a5b6-450eba9711cc" (UID: "2eaee46c-23ef-410a-a5b6-450eba9711cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.121113 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.120948    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53b2edae-97f1-44e9-9580-29d605357c8f-istiod-ca-cert" pod "53b2edae-97f1-44e9-9580-29d605357c8f" (UID: "53b2edae-97f1-44e9-9580-29d605357c8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:19.121896 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.121511    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-envoy" pod "2eaee46c-23ef-410a-a5b6-450eba9711cc" (UID: "2eaee46c-23ef-410a-a5b6-450eba9711cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.121896 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.121633    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b2edae-97f1-44e9-9580-29d605357c8f-istio-token" pod "53b2edae-97f1-44e9-9580-29d605357c8f" (UID: "53b2edae-97f1-44e9-9580-29d605357c8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:19.122058 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.121965    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eaee46c-23ef-410a-a5b6-450eba9711cc-istiod-ca-cert" pod "2eaee46c-23ef-410a-a5b6-450eba9711cc" (UID: "2eaee46c-23ef-410a-a5b6-450eba9711cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:19.122254 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.122199    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-workload-certs" pod "53b2edae-97f1-44e9-9580-29d605357c8f" (UID: "53b2edae-97f1-44e9-9580-29d605357c8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.125920 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.125879    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b2edae-97f1-44e9-9580-29d605357c8f-kube-api-access-5vwbp" pod "53b2edae-97f1-44e9-9580-29d605357c8f" (UID: "53b2edae-97f1-44e9-9580-29d605357c8f"). InnerVolumeSpecName "kube-api-access-5vwbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:19.127523 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.126041    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53b2edae-97f1-44e9-9580-29d605357c8f-istio-podinfo" pod "53b2edae-97f1-44e9-9580-29d605357c8f" (UID: "53b2edae-97f1-44e9-9580-29d605357c8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:09:19.127835 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.127746    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-token" pod "2eaee46c-23ef-410a-a5b6-450eba9711cc" (UID: "2eaee46c-23ef-410a-a5b6-450eba9711cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:19.127835 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.127809    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-podinfo" pod "2eaee46c-23ef-410a-a5b6-450eba9711cc" (UID: "2eaee46c-23ef-410a-a5b6-450eba9711cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:09:19.130222 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.130165    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-istio-envoy" pod "53b2edae-97f1-44e9-9580-29d605357c8f" (UID: "53b2edae-97f1-44e9-9580-29d605357c8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.131973 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.131950    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eaee46c-23ef-410a-a5b6-450eba9711cc-kube-api-access-zs7z9" pod "2eaee46c-23ef-410a-a5b6-450eba9711cc" (UID: "2eaee46c-23ef-410a-a5b6-450eba9711cc"). InnerVolumeSpecName "kube-api-access-zs7z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:19.166797 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.166776    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 19:09:19.221334 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.220878    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-workload-certs\") pod \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") "
Oct 30 19:09:19.221334 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.221093    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istiod-ca-cert\") pod \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") "
Oct 30 19:09:19.221334 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.221147    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-credential-socket\") pod \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") "
Oct 30 19:09:19.221334 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.221222    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-workload-certs" pod "34dfd0fe-b500-4df5-b191-6085a5d8dde5" (UID: "34dfd0fe-b500-4df5-b191-6085a5d8dde5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.221334 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.221278    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-envoy\") pod \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") "
Oct 30 19:09:19.221334 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.221318    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-podinfo\") pod \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") "
Oct 30 19:09:19.221792 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.221510    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-token\" (UniqueName: \"kubernetes.io/projected/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-token\") pod \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") "
Oct 30 19:09:19.221792 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.221542    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-data\") pod \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") "
Oct 30 19:09:19.221792 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.221572    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istiod-ca-cert" pod "34dfd0fe-b500-4df5-b191-6085a5d8dde5" (UID: "34dfd0fe-b500-4df5-b191-6085a5d8dde5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:19.221792 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.221617    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34dfd0fe-b500-4df5-b191-6085a5d8dde5-kube-api-access-lzjth\" (UniqueName: \"kubernetes.io/projected/34dfd0fe-b500-4df5-b191-6085a5d8dde5-kube-api-access-lzjth\") pod \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") "
Oct 30 19:09:19.221792 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.221727    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-workload-socket\") pod \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\" (UID: \"34dfd0fe-b500-4df5-b191-6085a5d8dde5\") "
Oct 30 19:09:19.229099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.222165    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-credential-socket" pod "34dfd0fe-b500-4df5-b191-6085a5d8dde5" (UID: "34dfd0fe-b500-4df5-b191-6085a5d8dde5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.229099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.222377    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-workload-socket" pod "34dfd0fe-b500-4df5-b191-6085a5d8dde5" (UID: "34dfd0fe-b500-4df5-b191-6085a5d8dde5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.229099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.222633    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-data" pod "34dfd0fe-b500-4df5-b191-6085a5d8dde5" (UID: "34dfd0fe-b500-4df5-b191-6085a5d8dde5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.229099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.222913    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.229099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.222936    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.229099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.222950    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53b2edae-97f1-44e9-9580-29d605357c8f-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.229099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.222964    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.229099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.222978    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.229099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.222991    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.229099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.223007    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.229099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.223020    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.229099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.223035    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.229099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.223052    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vwbp\" (UniqueName: \"kubernetes.io/projected/53b2edae-97f1-44e9-9580-29d605357c8f-kube-api-access-5vwbp\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.229099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.223165    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53b2edae-97f1-44e9-9580-29d605357c8f-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.229099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.223182    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eaee46c-23ef-410a-a5b6-450eba9711cc-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.229099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.223197    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs7z9\" (UniqueName: \"kubernetes.io/projected/2eaee46c-23ef-410a-a5b6-450eba9711cc-kube-api-access-zs7z9\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.229099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.223214    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eaee46c-23ef-410a-a5b6-450eba9711cc-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.229099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.223228    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53b2edae-97f1-44e9-9580-29d605357c8f-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.229099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.223242    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53b2edae-97f1-44e9-9580-29d605357c8f-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.229099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.223256    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.229099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.226370    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34dfd0fe-b500-4df5-b191-6085a5d8dde5-kube-api-access-lzjth" pod "34dfd0fe-b500-4df5-b191-6085a5d8dde5" (UID: "34dfd0fe-b500-4df5-b191-6085a5d8dde5"). InnerVolumeSpecName "kube-api-access-lzjth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:19.229099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.227612    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-podinfo" pod "34dfd0fe-b500-4df5-b191-6085a5d8dde5" (UID: "34dfd0fe-b500-4df5-b191-6085a5d8dde5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:09:19.229099 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.227999    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-envoy" pod "34dfd0fe-b500-4df5-b191-6085a5d8dde5" (UID: "34dfd0fe-b500-4df5-b191-6085a5d8dde5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.230206 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.229733    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-token" pod "34dfd0fe-b500-4df5-b191-6085a5d8dde5" (UID: "34dfd0fe-b500-4df5-b191-6085a5d8dde5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:19.231089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.231055    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:09:19.324793 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.324757    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-envoy\") pod \"922dfc11-a4ba-4d50-a50f-b5155d277792\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") "
Oct 30 19:09:19.324986 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.324828    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d743094-1851-4c3a-9866-ecb1062c69b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d743094-1851-4c3a-9866-ecb1062c69b6-istiod-ca-cert\") pod \"7d743094-1851-4c3a-9866-ecb1062c69b6\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") "
Oct 30 19:09:19.324986 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.324863    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-envoy\") pod \"7d743094-1851-4c3a-9866-ecb1062c69b6\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") "
Oct 30 19:09:19.324986 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.324969    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-podinfo\") pod \"922dfc11-a4ba-4d50-a50f-b5155d277792\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") "
Oct 30 19:09:19.325209 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.325019    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-workload-socket\") pod \"922dfc11-a4ba-4d50-a50f-b5155d277792\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") "
Oct 30 19:09:19.325209 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.325089    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-data\") pod \"922dfc11-a4ba-4d50-a50f-b5155d277792\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") "
Oct 30 19:09:19.325209 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.325139    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-workload-certs\") pod \"7d743094-1851-4c3a-9866-ecb1062c69b6\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") "
Oct 30 19:09:19.325449 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.325221    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d743094-1851-4c3a-9866-ecb1062c69b6-istiod-ca-cert" pod "7d743094-1851-4c3a-9866-ecb1062c69b6" (UID: "7d743094-1851-4c3a-9866-ecb1062c69b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:19.325449 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.325261    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-credential-socket\") pod \"7d743094-1851-4c3a-9866-ecb1062c69b6\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") "
Oct 30 19:09:19.325449 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.325293    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-workload-certs\") pod \"922dfc11-a4ba-4d50-a50f-b5155d277792\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") "
Oct 30 19:09:19.325449 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.325376    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-data\") pod \"7d743094-1851-4c3a-9866-ecb1062c69b6\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") "
Oct 30 19:09:19.325449 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.325420    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-podinfo\") pod \"7d743094-1851-4c3a-9866-ecb1062c69b6\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") "
Oct 30 19:09:19.325727 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.325509    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-credential-socket\") pod \"922dfc11-a4ba-4d50-a50f-b5155d277792\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") "
Oct 30 19:09:19.325727 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.325600    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d743094-1851-4c3a-9866-ecb1062c69b6-kube-api-access-bzqx9\" (UniqueName: \"kubernetes.io/projected/7d743094-1851-4c3a-9866-ecb1062c69b6-kube-api-access-bzqx9\") pod \"7d743094-1851-4c3a-9866-ecb1062c69b6\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") "
Oct 30 19:09:19.325727 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.325656    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/922dfc11-a4ba-4d50-a50f-b5155d277792-kube-api-access-9mchj\" (UniqueName: \"kubernetes.io/projected/922dfc11-a4ba-4d50-a50f-b5155d277792-kube-api-access-9mchj\") pod \"922dfc11-a4ba-4d50-a50f-b5155d277792\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") "
Oct 30 19:09:19.325911 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.325755    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/922dfc11-a4ba-4d50-a50f-b5155d277792-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922dfc11-a4ba-4d50-a50f-b5155d277792-istiod-ca-cert\") pod \"922dfc11-a4ba-4d50-a50f-b5155d277792\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") "
Oct 30 19:09:19.325911 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.325821    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-workload-socket\") pod \"7d743094-1851-4c3a-9866-ecb1062c69b6\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") "
Oct 30 19:09:19.325911 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.325848    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-token\" (UniqueName: \"kubernetes.io/projected/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-token\") pod \"922dfc11-a4ba-4d50-a50f-b5155d277792\" (UID: \"922dfc11-a4ba-4d50-a50f-b5155d277792\") "
Oct 30 19:09:19.325911 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.325906    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-token\" (UniqueName: \"kubernetes.io/projected/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-token\") pod \"7d743094-1851-4c3a-9866-ecb1062c69b6\" (UID: \"7d743094-1851-4c3a-9866-ecb1062c69b6\") "
Oct 30 19:09:19.326543 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.325510    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-workload-socket" pod "922dfc11-a4ba-4d50-a50f-b5155d277792" (UID: "922dfc11-a4ba-4d50-a50f-b5155d277792"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.326994 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.326966    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-workload-socket" pod "7d743094-1851-4c3a-9866-ecb1062c69b6" (UID: "7d743094-1851-4c3a-9866-ecb1062c69b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.327391 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.327365    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/922dfc11-a4ba-4d50-a50f-b5155d277792-istiod-ca-cert" pod "922dfc11-a4ba-4d50-a50f-b5155d277792" (UID: "922dfc11-a4ba-4d50-a50f-b5155d277792"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:19.327512 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.327482    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-envoy" pod "922dfc11-a4ba-4d50-a50f-b5155d277792" (UID: "922dfc11-a4ba-4d50-a50f-b5155d277792"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.327686 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.327663    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-credential-socket" pod "7d743094-1851-4c3a-9866-ecb1062c69b6" (UID: "7d743094-1851-4c3a-9866-ecb1062c69b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.328282 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.327828    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-podinfo" pod "922dfc11-a4ba-4d50-a50f-b5155d277792" (UID: "922dfc11-a4ba-4d50-a50f-b5155d277792"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:09:19.328282 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.327859    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-data" pod "922dfc11-a4ba-4d50-a50f-b5155d277792" (UID: "922dfc11-a4ba-4d50-a50f-b5155d277792"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.328282 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.328011    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-envoy" pod "7d743094-1851-4c3a-9866-ecb1062c69b6" (UID: "7d743094-1851-4c3a-9866-ecb1062c69b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.328282 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.328057    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-workload-certs" pod "7d743094-1851-4c3a-9866-ecb1062c69b6" (UID: "7d743094-1851-4c3a-9866-ecb1062c69b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.328282 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.328114    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-data" pod "7d743094-1851-4c3a-9866-ecb1062c69b6" (UID: "7d743094-1851-4c3a-9866-ecb1062c69b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.328282 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.328200    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-credential-socket" pod "922dfc11-a4ba-4d50-a50f-b5155d277792" (UID: "922dfc11-a4ba-4d50-a50f-b5155d277792"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.329622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.328611    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-workload-certs" pod "922dfc11-a4ba-4d50-a50f-b5155d277792" (UID: "922dfc11-a4ba-4d50-a50f-b5155d277792"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:19.329622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.329156    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.329622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.329178    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.329622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.329193    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzjth\" (UniqueName: \"kubernetes.io/projected/34dfd0fe-b500-4df5-b191-6085a5d8dde5-kube-api-access-lzjth\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.329622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.329208    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.329622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.329224    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.329622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.329239    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.329622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.329253    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.329622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.329267    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922dfc11-a4ba-4d50-a50f-b5155d277792-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.329622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.329280    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.329622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.329294    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.329622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.329308    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34dfd0fe-b500-4df5-b191-6085a5d8dde5-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.329622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.329321    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.329622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.329334    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d743094-1851-4c3a-9866-ecb1062c69b6-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.329622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.329347    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.329622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.329362    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.329622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.329377    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.329622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.329393    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.330623 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.329694    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-token" pod "922dfc11-a4ba-4d50-a50f-b5155d277792" (UID: "922dfc11-a4ba-4d50-a50f-b5155d277792"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:19.330623 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.330348    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-podinfo" pod "7d743094-1851-4c3a-9866-ecb1062c69b6" (UID: "7d743094-1851-4c3a-9866-ecb1062c69b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:09:19.330623 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.330443    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922dfc11-a4ba-4d50-a50f-b5155d277792-kube-api-access-9mchj" pod "922dfc11-a4ba-4d50-a50f-b5155d277792" (UID: "922dfc11-a4ba-4d50-a50f-b5155d277792"). InnerVolumeSpecName "kube-api-access-9mchj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:19.330623 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.330549    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d743094-1851-4c3a-9866-ecb1062c69b6-kube-api-access-bzqx9" pod "7d743094-1851-4c3a-9866-ecb1062c69b6" (UID: "7d743094-1851-4c3a-9866-ecb1062c69b6"). InnerVolumeSpecName "kube-api-access-bzqx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:19.331697 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.331676    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-token" pod "7d743094-1851-4c3a-9866-ecb1062c69b6" (UID: "7d743094-1851-4c3a-9866-ecb1062c69b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:19.431297 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.431216    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.431297 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.431252    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzqx9\" (UniqueName: \"kubernetes.io/projected/7d743094-1851-4c3a-9866-ecb1062c69b6-kube-api-access-bzqx9\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.431297 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.431270    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mchj\" (UniqueName: \"kubernetes.io/projected/922dfc11-a4ba-4d50-a50f-b5155d277792-kube-api-access-9mchj\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.431297 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.431283    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922dfc11-a4ba-4d50-a50f-b5155d277792-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.431297 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.431298    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d743094-1851-4c3a-9866-ecb1062c69b6-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:19.616292 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.616257    4924 generic.go:356] "Generic (PLEG): container finished" podID="7d743094-1851-4c3a-9866-ecb1062c69b6" containerID="d898f5123e81a57cd207d07ece648447a1cb0b5e632a8f96ce1333f52d89d449" exitCode=0
Oct 30 19:09:19.616571 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.616327    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk" event={"ID":"7d743094-1851-4c3a-9866-ecb1062c69b6","Type":"ContainerDied","Data":"d898f5123e81a57cd207d07ece648447a1cb0b5e632a8f96ce1333f52d89d449"}
Oct 30 19:09:19.616571 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.616393    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk" event={"ID":"7d743094-1851-4c3a-9866-ecb1062c69b6","Type":"ContainerDied","Data":"ad409711193beda63574b25f5581d1d6069e3397fca3a76bd136570e326e0165"}
Oct 30 19:09:19.616571 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.616511    4924 scope.go:122] "RemoveContainer" containerID="d898f5123e81a57cd207d07ece648447a1cb0b5e632a8f96ce1333f52d89d449"
Oct 30 19:09:19.617055 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.617033    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"
Oct 30 19:09:19.619565 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.619531    4924 generic.go:356] "Generic (PLEG): container finished" podID="34dfd0fe-b500-4df5-b191-6085a5d8dde5" containerID="6a7aa424319bce6c1197a1f465f648982b85daafa1d99bfbb22483f3a46178df" exitCode=0
Oct 30 19:09:19.619683 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.619664    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2" event={"ID":"34dfd0fe-b500-4df5-b191-6085a5d8dde5","Type":"ContainerDied","Data":"6a7aa424319bce6c1197a1f465f648982b85daafa1d99bfbb22483f3a46178df"}
Oct 30 19:09:19.619763 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.619710    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2" event={"ID":"34dfd0fe-b500-4df5-b191-6085a5d8dde5","Type":"ContainerDied","Data":"cd580ed539c16295dbf7466b16eeaf0eeea057f15bc2cf73f95867a064450445"}
Oct 30 19:09:19.619839 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.619791    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"
Oct 30 19:09:19.639313 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.638574    4924 generic.go:356] "Generic (PLEG): container finished" podID="922dfc11-a4ba-4d50-a50f-b5155d277792" containerID="51132c3f0c9a06d65bd4696f29e0ad1db4388b8946da5d03eb28807462cef97c" exitCode=0
Oct 30 19:09:19.639313 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.638739    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw" event={"ID":"922dfc11-a4ba-4d50-a50f-b5155d277792","Type":"ContainerDied","Data":"51132c3f0c9a06d65bd4696f29e0ad1db4388b8946da5d03eb28807462cef97c"}
Oct 30 19:09:19.639313 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.638787    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw" event={"ID":"922dfc11-a4ba-4d50-a50f-b5155d277792","Type":"ContainerDied","Data":"0e92a32e11eb36b4dd02bcf8a9277b0111623f1dfe4c21dd8fc9682d4938d8cf"}
Oct 30 19:09:19.639313 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.638868    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"
Oct 30 19:09:19.641133 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.641109    4924 generic.go:356] "Generic (PLEG): container finished" podID="2eaee46c-23ef-410a-a5b6-450eba9711cc" containerID="b82e787d0d3b24eff24c9b84268fd81e2d3db47b25dbffb04823654bdf6863ca" exitCode=0
Oct 30 19:09:19.641246 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.641228    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p" event={"ID":"2eaee46c-23ef-410a-a5b6-450eba9711cc","Type":"ContainerDied","Data":"b82e787d0d3b24eff24c9b84268fd81e2d3db47b25dbffb04823654bdf6863ca"}
Oct 30 19:09:19.641326 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.641271    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p" event={"ID":"2eaee46c-23ef-410a-a5b6-450eba9711cc","Type":"ContainerDied","Data":"2269e6f93fddeff5fd8aca8fc707cbf6ba3a758789678f52ec66286bde0f0786"}
Oct 30 19:09:19.641390 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.641345    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"
Oct 30 19:09:19.643787 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.643762    4924 generic.go:356] "Generic (PLEG): container finished" podID="53b2edae-97f1-44e9-9580-29d605357c8f" containerID="5ba0865c313b090504e1e17e13cfe38c45726c245351ba777b4d11feb53c5cb8" exitCode=0
Oct 30 19:09:19.643907 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.643852    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx" event={"ID":"53b2edae-97f1-44e9-9580-29d605357c8f","Type":"ContainerDied","Data":"5ba0865c313b090504e1e17e13cfe38c45726c245351ba777b4d11feb53c5cb8"}
Oct 30 19:09:19.643907 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.643889    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx" event={"ID":"53b2edae-97f1-44e9-9580-29d605357c8f","Type":"ContainerDied","Data":"1f30d3d9714a9cbb4127955dcfea4f0d74b430d9afdcd44f2c8df3f529dc5c0d"}
Oct 30 19:09:19.644023 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.643955    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"
Oct 30 19:09:19.656866 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.656850    4924 scope.go:122] "RemoveContainer" containerID="d898f5123e81a57cd207d07ece648447a1cb0b5e632a8f96ce1333f52d89d449"
Oct 30 19:09:19.657262 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:09:19.657231    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d898f5123e81a57cd207d07ece648447a1cb0b5e632a8f96ce1333f52d89d449\": container with ID starting with d898f5123e81a57cd207d07ece648447a1cb0b5e632a8f96ce1333f52d89d449 not found: ID does not exist" containerID="d898f5123e81a57cd207d07ece648447a1cb0b5e632a8f96ce1333f52d89d449"
Oct 30 19:09:19.657385 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.657287    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d898f5123e81a57cd207d07ece648447a1cb0b5e632a8f96ce1333f52d89d449"} err="failed to get container status \"d898f5123e81a57cd207d07ece648447a1cb0b5e632a8f96ce1333f52d89d449\": rpc error: code = NotFound desc = could not find container \"d898f5123e81a57cd207d07ece648447a1cb0b5e632a8f96ce1333f52d89d449\": container with ID starting with d898f5123e81a57cd207d07ece648447a1cb0b5e632a8f96ce1333f52d89d449 not found: ID does not exist"
Oct 30 19:09:19.657385 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.657311    4924 scope.go:122] "RemoveContainer" containerID="6a7aa424319bce6c1197a1f465f648982b85daafa1d99bfbb22483f3a46178df"
Oct 30 19:09:19.678417 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.678393    4924 scope.go:122] "RemoveContainer" containerID="6a7aa424319bce6c1197a1f465f648982b85daafa1d99bfbb22483f3a46178df"
Oct 30 19:09:19.678904 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:09:19.678882    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a7aa424319bce6c1197a1f465f648982b85daafa1d99bfbb22483f3a46178df\": container with ID starting with 6a7aa424319bce6c1197a1f465f648982b85daafa1d99bfbb22483f3a46178df not found: ID does not exist" containerID="6a7aa424319bce6c1197a1f465f648982b85daafa1d99bfbb22483f3a46178df"
Oct 30 19:09:19.679011 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.678914    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a7aa424319bce6c1197a1f465f648982b85daafa1d99bfbb22483f3a46178df"} err="failed to get container status \"6a7aa424319bce6c1197a1f465f648982b85daafa1d99bfbb22483f3a46178df\": rpc error: code = NotFound desc = could not find container \"6a7aa424319bce6c1197a1f465f648982b85daafa1d99bfbb22483f3a46178df\": container with ID starting with 6a7aa424319bce6c1197a1f465f648982b85daafa1d99bfbb22483f3a46178df not found: ID does not exist"
Oct 30 19:09:19.679011 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.678939    4924 scope.go:122] "RemoveContainer" containerID="51132c3f0c9a06d65bd4696f29e0ad1db4388b8946da5d03eb28807462cef97c"
Oct 30 19:09:19.688878 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.688846    4924 scope.go:122] "RemoveContainer" containerID="51132c3f0c9a06d65bd4696f29e0ad1db4388b8946da5d03eb28807462cef97c"
Oct 30 19:09:19.689217 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:09:19.689195    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51132c3f0c9a06d65bd4696f29e0ad1db4388b8946da5d03eb28807462cef97c\": container with ID starting with 51132c3f0c9a06d65bd4696f29e0ad1db4388b8946da5d03eb28807462cef97c not found: ID does not exist" containerID="51132c3f0c9a06d65bd4696f29e0ad1db4388b8946da5d03eb28807462cef97c"
Oct 30 19:09:19.689315 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.689227    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51132c3f0c9a06d65bd4696f29e0ad1db4388b8946da5d03eb28807462cef97c"} err="failed to get container status \"51132c3f0c9a06d65bd4696f29e0ad1db4388b8946da5d03eb28807462cef97c\": rpc error: code = NotFound desc = could not find container \"51132c3f0c9a06d65bd4696f29e0ad1db4388b8946da5d03eb28807462cef97c\": container with ID starting with 51132c3f0c9a06d65bd4696f29e0ad1db4388b8946da5d03eb28807462cef97c not found: ID does not exist"
Oct 30 19:09:19.689315 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.689253    4924 scope.go:122] "RemoveContainer" containerID="b82e787d0d3b24eff24c9b84268fd81e2d3db47b25dbffb04823654bdf6863ca"
Oct 30 19:09:19.698610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.698591    4924 scope.go:122] "RemoveContainer" containerID="b82e787d0d3b24eff24c9b84268fd81e2d3db47b25dbffb04823654bdf6863ca"
Oct 30 19:09:19.698903 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:09:19.698869    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b82e787d0d3b24eff24c9b84268fd81e2d3db47b25dbffb04823654bdf6863ca\": container with ID starting with b82e787d0d3b24eff24c9b84268fd81e2d3db47b25dbffb04823654bdf6863ca not found: ID does not exist" containerID="b82e787d0d3b24eff24c9b84268fd81e2d3db47b25dbffb04823654bdf6863ca"
Oct 30 19:09:19.698964 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.698913    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b82e787d0d3b24eff24c9b84268fd81e2d3db47b25dbffb04823654bdf6863ca"} err="failed to get container status \"b82e787d0d3b24eff24c9b84268fd81e2d3db47b25dbffb04823654bdf6863ca\": rpc error: code = NotFound desc = could not find container \"b82e787d0d3b24eff24c9b84268fd81e2d3db47b25dbffb04823654bdf6863ca\": container with ID starting with b82e787d0d3b24eff24c9b84268fd81e2d3db47b25dbffb04823654bdf6863ca not found: ID does not exist"
Oct 30 19:09:19.698964 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.698937    4924 scope.go:122] "RemoveContainer" containerID="5ba0865c313b090504e1e17e13cfe38c45726c245351ba777b4d11feb53c5cb8"
Oct 30 19:09:19.708553 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.708533    4924 scope.go:122] "RemoveContainer" containerID="5ba0865c313b090504e1e17e13cfe38c45726c245351ba777b4d11feb53c5cb8"
Oct 30 19:09:19.708835 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:09:19.708818    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ba0865c313b090504e1e17e13cfe38c45726c245351ba777b4d11feb53c5cb8\": container with ID starting with 5ba0865c313b090504e1e17e13cfe38c45726c245351ba777b4d11feb53c5cb8 not found: ID does not exist" containerID="5ba0865c313b090504e1e17e13cfe38c45726c245351ba777b4d11feb53c5cb8"
Oct 30 19:09:19.708893 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.708842    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ba0865c313b090504e1e17e13cfe38c45726c245351ba777b4d11feb53c5cb8"} err="failed to get container status \"5ba0865c313b090504e1e17e13cfe38c45726c245351ba777b4d11feb53c5cb8\": rpc error: code = NotFound desc = could not find container \"5ba0865c313b090504e1e17e13cfe38c45726c245351ba777b4d11feb53c5cb8\": container with ID starting with 5ba0865c313b090504e1e17e13cfe38c45726c245351ba777b4d11feb53c5cb8 not found: ID does not exist"
Oct 30 19:09:19.763992 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.763947    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:19.911178 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.911145    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"]
Oct 30 19:09:19.914790 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:19.914767    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-tp4kk"]
Oct 30 19:09:20.026714 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:20.026621    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"]
Oct 30 19:09:20.030925 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:20.030896    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-qzr6p"]
Oct 30 19:09:20.132893 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:20.132861    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"]
Oct 30 19:09:20.137365 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:20.137339    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-ctqlx"]
Oct 30 19:09:20.139399 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:20.139366    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:20.243698 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:20.243665    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"]
Oct 30 19:09:20.249107 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:20.249082    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-wplbw"]
Oct 30 19:09:20.358567 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:20.358532    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"]
Oct 30 19:09:20.362416 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:20.362388    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-rsmj2"]
Oct 30 19:09:20.836876 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:20.836799    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eaee46c-23ef-410a-a5b6-450eba9711cc" path="/var/lib/kubelet/pods/2eaee46c-23ef-410a-a5b6-450eba9711cc/volumes"
Oct 30 19:09:20.837430 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:20.837403    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34dfd0fe-b500-4df5-b191-6085a5d8dde5" path="/var/lib/kubelet/pods/34dfd0fe-b500-4df5-b191-6085a5d8dde5/volumes"
Oct 30 19:09:20.838040 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:20.838019    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53b2edae-97f1-44e9-9580-29d605357c8f" path="/var/lib/kubelet/pods/53b2edae-97f1-44e9-9580-29d605357c8f/volumes"
Oct 30 19:09:20.839186 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:20.839159    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d743094-1851-4c3a-9866-ecb1062c69b6" path="/var/lib/kubelet/pods/7d743094-1851-4c3a-9866-ecb1062c69b6/volumes"
Oct 30 19:09:20.839787 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:20.839771    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="922dfc11-a4ba-4d50-a50f-b5155d277792" path="/var/lib/kubelet/pods/922dfc11-a4ba-4d50-a50f-b5155d277792/volumes"
Oct 30 19:09:23.308820 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:23.308758    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:23.514496 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:23.514460    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:23.604969 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:23.604926    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"]
Oct 30 19:09:23.605205 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:23.605146    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq" podUID="1706cc50-6d51-4e2a-8cf1-308f6e2187b0" containerName="discovery" containerID="cri-o://090740e26b8a8fcab79cd332873d6dc800f14c33bf5f7ec93340edf1a5fa453a" gracePeriod=30
Oct 30 19:09:23.900490 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:23.900469    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:09:24.014495 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.014464    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-csr-ca-configmap\") pod \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") "
Oct 30 19:09:24.014670 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.014534    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-token\" (UniqueName: \"kubernetes.io/projected/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-token\") pod \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") "
Oct 30 19:09:24.014670 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.014564    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-cacerts\" (UniqueName: \"kubernetes.io/secret/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-cacerts\") pod \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") "
Oct 30 19:09:24.014670 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.014637    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-kubeconfig\") pod \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") "
Oct 30 19:09:24.014670 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.014664    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-local-certs\") pod \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") "
Oct 30 19:09:24.015033 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.014697    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-kube-api-access-4csxk\" (UniqueName: \"kubernetes.io/projected/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-kube-api-access-4csxk\") pod \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") "
Oct 30 19:09:24.015033 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.014810    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-csr-dns-cert\") pod \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\" (UID: \"1706cc50-6d51-4e2a-8cf1-308f6e2187b0\") "
Oct 30 19:09:24.015033 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.014885    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-csr-ca-configmap" pod "1706cc50-6d51-4e2a-8cf1-308f6e2187b0" (UID: "1706cc50-6d51-4e2a-8cf1-308f6e2187b0"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:24.015544 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.015496    4924 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-csr-ca-configmap\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:24.017781 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.017724    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-kubeconfig" pod "1706cc50-6d51-4e2a-8cf1-308f6e2187b0" (UID: "1706cc50-6d51-4e2a-8cf1-308f6e2187b0"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:09:24.017781 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.017747    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-cacerts" pod "1706cc50-6d51-4e2a-8cf1-308f6e2187b0" (UID: "1706cc50-6d51-4e2a-8cf1-308f6e2187b0"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:09:24.017781 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.017750    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-kube-api-access-4csxk" pod "1706cc50-6d51-4e2a-8cf1-308f6e2187b0" (UID: "1706cc50-6d51-4e2a-8cf1-308f6e2187b0"). InnerVolumeSpecName "kube-api-access-4csxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:24.017781 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.017771    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-token" pod "1706cc50-6d51-4e2a-8cf1-308f6e2187b0" (UID: "1706cc50-6d51-4e2a-8cf1-308f6e2187b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:24.020463 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.020413    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-csr-dns-cert" pod "1706cc50-6d51-4e2a-8cf1-308f6e2187b0" (UID: "1706cc50-6d51-4e2a-8cf1-308f6e2187b0"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:09:24.022336 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.022314    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-local-certs" pod "1706cc50-6d51-4e2a-8cf1-308f6e2187b0" (UID: "1706cc50-6d51-4e2a-8cf1-308f6e2187b0"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:24.117826 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.117738    4924 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-csr-dns-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:24.117826 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.117776    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:24.117826 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.117791    4924 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-cacerts\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:24.117826 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.117805    4924 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-istio-kubeconfig\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:24.118116 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.117842    4924 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-local-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:24.118116 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.117879    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4csxk\" (UniqueName: \"kubernetes.io/projected/1706cc50-6d51-4e2a-8cf1-308f6e2187b0-kube-api-access-4csxk\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:24.697458 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.697417    4924 generic.go:356] "Generic (PLEG): container finished" podID="1706cc50-6d51-4e2a-8cf1-308f6e2187b0" containerID="090740e26b8a8fcab79cd332873d6dc800f14c33bf5f7ec93340edf1a5fa453a" exitCode=0
Oct 30 19:09:24.697889 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.697481    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"
Oct 30 19:09:24.697889 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.697497    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq" event={"ID":"1706cc50-6d51-4e2a-8cf1-308f6e2187b0","Type":"ContainerDied","Data":"090740e26b8a8fcab79cd332873d6dc800f14c33bf5f7ec93340edf1a5fa453a"}
Oct 30 19:09:24.697889 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.697584    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq" event={"ID":"1706cc50-6d51-4e2a-8cf1-308f6e2187b0","Type":"ContainerDied","Data":"9ae9fb2eaff5e923317709b1fe22c7e7b76f250843c2b4e989592ba166e31b9c"}
Oct 30 19:09:24.697889 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.697608    4924 scope.go:122] "RemoveContainer" containerID="090740e26b8a8fcab79cd332873d6dc800f14c33bf5f7ec93340edf1a5fa453a"
Oct 30 19:09:24.709505 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.709485    4924 scope.go:122] "RemoveContainer" containerID="090740e26b8a8fcab79cd332873d6dc800f14c33bf5f7ec93340edf1a5fa453a"
Oct 30 19:09:24.709749 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:09:24.709732    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"090740e26b8a8fcab79cd332873d6dc800f14c33bf5f7ec93340edf1a5fa453a\": container with ID starting with 090740e26b8a8fcab79cd332873d6dc800f14c33bf5f7ec93340edf1a5fa453a not found: ID does not exist" containerID="090740e26b8a8fcab79cd332873d6dc800f14c33bf5f7ec93340edf1a5fa453a"
Oct 30 19:09:24.709791 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.709757    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"090740e26b8a8fcab79cd332873d6dc800f14c33bf5f7ec93340edf1a5fa453a"} err="failed to get container status \"090740e26b8a8fcab79cd332873d6dc800f14c33bf5f7ec93340edf1a5fa453a\": rpc error: code = NotFound desc = could not find container \"090740e26b8a8fcab79cd332873d6dc800f14c33bf5f7ec93340edf1a5fa453a\": container with ID starting with 090740e26b8a8fcab79cd332873d6dc800f14c33bf5f7ec93340edf1a5fa453a not found: ID does not exist"
Oct 30 19:09:24.902921 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.902883    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"]
Oct 30 19:09:24.949212 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:24.949133    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7t6jq"]
Oct 30 19:09:26.836113 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:26.836081    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1706cc50-6d51-4e2a-8cf1-308f6e2187b0" path="/var/lib/kubelet/pods/1706cc50-6d51-4e2a-8cf1-308f6e2187b0/volumes"
Oct 30 19:09:27.048842 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:27.048808    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fa834f9-bwffr"]
Oct 30 19:09:27.095294 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:27.095212    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fa834f9-bwffr"]
Oct 30 19:09:28.316970 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:28.316936    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:28.836652 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:28.836618    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48e2de76-33fe-4942-ac44-5ea1d92d8bde" path="/var/lib/kubelet/pods/48e2de76-33fe-4942-ac44-5ea1d92d8bde/volumes"
Oct 30 19:09:30.298528 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:30.298494    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"]
Oct 30 19:09:30.299100 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:30.298848    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4" podUID="4c5b2946-7818-4067-976a-7e08e0f5c66c" containerName="istio-proxy" containerID="cri-o://06577ab5d460dc9cb1304d2c129618e503d42c52319d27061f982c77bf0b8e6a" gracePeriod=30
Oct 30 19:09:30.345591 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:30.345558    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"]
Oct 30 19:09:30.345858 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:30.345826    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964" podUID="dc3b282b-17c0-4f77-ba43-540f9524a8a9" containerName="istio-proxy" containerID="cri-o://e0390dba402579c3422fdd1fcc4161e2162c4343de4e101656fcd2f3349f467a" gracePeriod=30
Oct 30 19:09:30.349462 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:30.349430    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"]
Oct 30 19:09:30.349655 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:30.349634    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g" podUID="f53aa846-ec01-4b44-a518-49a558965906" containerName="istio-proxy" containerID="cri-o://01ad1b91e81af2696711f1b54ce954f9b675972822cdd80c2a184acf8a10209b" gracePeriod=30
Oct 30 19:09:30.392012 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:30.391986    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"]
Oct 30 19:09:30.392200 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:30.392182    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2" podUID="fa7b7f06-28aa-4a1f-ac1c-d1d363815699" containerName="istio-proxy" containerID="cri-o://30d947932b02122854a92acc189c210cd0e0bdcbf032836c9f5dd7fddcb3afca" gracePeriod=30
Oct 30 19:09:33.320702 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:33.320647    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:34.737329 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:34.737292    4924 patch_prober.go:29] interesting pod/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:09:34.737722 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:34.737364    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2" podUID="fa7b7f06-28aa-4a1f-ac1c-d1d363815699" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:09:35.648033 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.647913    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:09:35.739782 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.739755    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:09:35.802086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.796356    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c5b2946-7818-4067-976a-7e08e0f5c66c-kube-api-access-6w2v9\" (UniqueName: \"kubernetes.io/projected/4c5b2946-7818-4067-976a-7e08e0f5c66c-kube-api-access-6w2v9\") pod \"4c5b2946-7818-4067-976a-7e08e0f5c66c\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") "
Oct 30 19:09:35.802086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.796454    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-workload-certs\") pod \"4c5b2946-7818-4067-976a-7e08e0f5c66c\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") "
Oct 30 19:09:35.802086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.796608    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-envoy\") pod \"4c5b2946-7818-4067-976a-7e08e0f5c66c\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") "
Oct 30 19:09:35.802086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.796660    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-credential-socket\") pod \"4c5b2946-7818-4067-976a-7e08e0f5c66c\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") "
Oct 30 19:09:35.802086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.796725    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c5b2946-7818-4067-976a-7e08e0f5c66c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c5b2946-7818-4067-976a-7e08e0f5c66c-istiod-ca-cert\") pod \"4c5b2946-7818-4067-976a-7e08e0f5c66c\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") "
Oct 30 19:09:35.802086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.796810    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-workload-socket\") pod \"4c5b2946-7818-4067-976a-7e08e0f5c66c\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") "
Oct 30 19:09:35.802086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.796862    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-data\") pod \"4c5b2946-7818-4067-976a-7e08e0f5c66c\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") "
Oct 30 19:09:35.802086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.796908    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-podinfo\") pod \"4c5b2946-7818-4067-976a-7e08e0f5c66c\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") "
Oct 30 19:09:35.802086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.796958    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-token\" (UniqueName: \"kubernetes.io/projected/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-token\") pod \"4c5b2946-7818-4067-976a-7e08e0f5c66c\" (UID: \"4c5b2946-7818-4067-976a-7e08e0f5c66c\") "
Oct 30 19:09:35.802086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.800338    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-workload-certs" pod "4c5b2946-7818-4067-976a-7e08e0f5c66c" (UID: "4c5b2946-7818-4067-976a-7e08e0f5c66c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:35.802086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.800350    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-credential-socket" pod "4c5b2946-7818-4067-976a-7e08e0f5c66c" (UID: "4c5b2946-7818-4067-976a-7e08e0f5c66c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:35.802086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.800542    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-workload-socket" pod "4c5b2946-7818-4067-976a-7e08e0f5c66c" (UID: "4c5b2946-7818-4067-976a-7e08e0f5c66c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:35.802086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.800555    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-data" pod "4c5b2946-7818-4067-976a-7e08e0f5c66c" (UID: "4c5b2946-7818-4067-976a-7e08e0f5c66c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:35.802086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.800748    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c5b2946-7818-4067-976a-7e08e0f5c66c-istiod-ca-cert" pod "4c5b2946-7818-4067-976a-7e08e0f5c66c" (UID: "4c5b2946-7818-4067-976a-7e08e0f5c66c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:35.802086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.800834    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-token" pod "4c5b2946-7818-4067-976a-7e08e0f5c66c" (UID: "4c5b2946-7818-4067-976a-7e08e0f5c66c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:35.804745 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.803238    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c5b2946-7818-4067-976a-7e08e0f5c66c-kube-api-access-6w2v9" pod "4c5b2946-7818-4067-976a-7e08e0f5c66c" (UID: "4c5b2946-7818-4067-976a-7e08e0f5c66c"). InnerVolumeSpecName "kube-api-access-6w2v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:35.806044 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.806015    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-envoy" pod "4c5b2946-7818-4067-976a-7e08e0f5c66c" (UID: "4c5b2946-7818-4067-976a-7e08e0f5c66c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:35.806201 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.806033    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-podinfo" pod "4c5b2946-7818-4067-976a-7e08e0f5c66c" (UID: "4c5b2946-7818-4067-976a-7e08e0f5c66c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:09:35.807397 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.807369    4924 generic.go:356] "Generic (PLEG): container finished" podID="fa7b7f06-28aa-4a1f-ac1c-d1d363815699" containerID="30d947932b02122854a92acc189c210cd0e0bdcbf032836c9f5dd7fddcb3afca" exitCode=0
Oct 30 19:09:35.807539 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.807520    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2" event={"ID":"fa7b7f06-28aa-4a1f-ac1c-d1d363815699","Type":"ContainerDied","Data":"30d947932b02122854a92acc189c210cd0e0bdcbf032836c9f5dd7fddcb3afca"}
Oct 30 19:09:35.816655 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.816633    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:09:35.817193 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.817172    4924 generic.go:356] "Generic (PLEG): container finished" podID="4c5b2946-7818-4067-976a-7e08e0f5c66c" containerID="06577ab5d460dc9cb1304d2c129618e503d42c52319d27061f982c77bf0b8e6a" exitCode=0
Oct 30 19:09:35.817347 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.817330    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4" event={"ID":"4c5b2946-7818-4067-976a-7e08e0f5c66c","Type":"ContainerDied","Data":"06577ab5d460dc9cb1304d2c129618e503d42c52319d27061f982c77bf0b8e6a"}
Oct 30 19:09:35.817422 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.817379    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4" event={"ID":"4c5b2946-7818-4067-976a-7e08e0f5c66c","Type":"ContainerDied","Data":"cfe967d0dfa5ca75e00fd75aa0cd79c5ebe940a630fc0b5a2685dbb3aa9fac6c"}
Oct 30 19:09:35.817422 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.817418    4924 scope.go:122] "RemoveContainer" containerID="06577ab5d460dc9cb1304d2c129618e503d42c52319d27061f982c77bf0b8e6a"
Oct 30 19:09:35.817589 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.817574    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"
Oct 30 19:09:35.825667 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.825580    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 19:09:35.825844 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.825816    4924 generic.go:356] "Generic (PLEG): container finished" podID="f53aa846-ec01-4b44-a518-49a558965906" containerID="01ad1b91e81af2696711f1b54ce954f9b675972822cdd80c2a184acf8a10209b" exitCode=0
Oct 30 19:09:35.825941 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.825921    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g" event={"ID":"f53aa846-ec01-4b44-a518-49a558965906","Type":"ContainerDied","Data":"01ad1b91e81af2696711f1b54ce954f9b675972822cdd80c2a184acf8a10209b"}
Oct 30 19:09:35.828930 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.828902    4924 generic.go:356] "Generic (PLEG): container finished" podID="dc3b282b-17c0-4f77-ba43-540f9524a8a9" containerID="e0390dba402579c3422fdd1fcc4161e2162c4343de4e101656fcd2f3349f467a" exitCode=0
Oct 30 19:09:35.829172 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.829042    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964" event={"ID":"dc3b282b-17c0-4f77-ba43-540f9524a8a9","Type":"ContainerDied","Data":"e0390dba402579c3422fdd1fcc4161e2162c4343de4e101656fcd2f3349f467a"}
Oct 30 19:09:35.829241 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.829171    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964" event={"ID":"dc3b282b-17c0-4f77-ba43-540f9524a8a9","Type":"ContainerDied","Data":"fabfa151d09d2b894fbd27d1e3206509393e13d9771bb3c2d203172ce8509148"}
Oct 30 19:09:35.829468 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.829443    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"
Oct 30 19:09:35.841241 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.841180    4924 scope.go:122] "RemoveContainer" containerID="06577ab5d460dc9cb1304d2c129618e503d42c52319d27061f982c77bf0b8e6a"
Oct 30 19:09:35.841484 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:09:35.841464    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06577ab5d460dc9cb1304d2c129618e503d42c52319d27061f982c77bf0b8e6a\": container with ID starting with 06577ab5d460dc9cb1304d2c129618e503d42c52319d27061f982c77bf0b8e6a not found: ID does not exist" containerID="06577ab5d460dc9cb1304d2c129618e503d42c52319d27061f982c77bf0b8e6a"
Oct 30 19:09:35.841565 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.841491    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06577ab5d460dc9cb1304d2c129618e503d42c52319d27061f982c77bf0b8e6a"} err="failed to get container status \"06577ab5d460dc9cb1304d2c129618e503d42c52319d27061f982c77bf0b8e6a\": rpc error: code = NotFound desc = could not find container \"06577ab5d460dc9cb1304d2c129618e503d42c52319d27061f982c77bf0b8e6a\": container with ID starting with 06577ab5d460dc9cb1304d2c129618e503d42c52319d27061f982c77bf0b8e6a not found: ID does not exist"
Oct 30 19:09:35.841565 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.841513    4924 scope.go:122] "RemoveContainer" containerID="01ad1b91e81af2696711f1b54ce954f9b675972822cdd80c2a184acf8a10209b"
Oct 30 19:09:35.855996 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.855937    4924 scope.go:122] "RemoveContainer" containerID="b7ed64f2a0ffae646e5fd3f7e0d50c1ce291c58ad8d2678edbae7d7165eaf89a"
Oct 30 19:09:35.869035 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.869003    4924 scope.go:122] "RemoveContainer" containerID="e0390dba402579c3422fdd1fcc4161e2162c4343de4e101656fcd2f3349f467a"
Oct 30 19:09:35.887088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.886509    4924 scope.go:122] "RemoveContainer" containerID="e0390dba402579c3422fdd1fcc4161e2162c4343de4e101656fcd2f3349f467a"
Oct 30 19:09:35.887088 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:09:35.886599    4924 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c5b2946_7818_4067_976a_7e08e0f5c66c.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 19:09:35.887088 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:09:35.886914    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0390dba402579c3422fdd1fcc4161e2162c4343de4e101656fcd2f3349f467a\": container with ID starting with e0390dba402579c3422fdd1fcc4161e2162c4343de4e101656fcd2f3349f467a not found: ID does not exist" containerID="e0390dba402579c3422fdd1fcc4161e2162c4343de4e101656fcd2f3349f467a"
Oct 30 19:09:35.887088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.886948    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0390dba402579c3422fdd1fcc4161e2162c4343de4e101656fcd2f3349f467a"} err="failed to get container status \"e0390dba402579c3422fdd1fcc4161e2162c4343de4e101656fcd2f3349f467a\": rpc error: code = NotFound desc = could not find container \"e0390dba402579c3422fdd1fcc4161e2162c4343de4e101656fcd2f3349f467a\": container with ID starting with e0390dba402579c3422fdd1fcc4161e2162c4343de4e101656fcd2f3349f467a not found: ID does not exist"
Oct 30 19:09:35.899220 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.899195    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-workload-socket\") pod \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") "
Oct 30 19:09:35.899329 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.899275    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istiod-ca-cert\") pod \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") "
Oct 30 19:09:35.899399 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.899348    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-token\" (UniqueName: \"kubernetes.io/projected/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-token\") pod \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") "
Oct 30 19:09:35.899399 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.899386    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-data\") pod \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") "
Oct 30 19:09:35.899509 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.899414    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-envoy\") pod \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") "
Oct 30 19:09:35.899597 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.899581    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-podinfo\") pod \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") "
Oct 30 19:09:35.899663 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.899640    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-workload-certs\") pod \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") "
Oct 30 19:09:35.899728 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.899713    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc3b282b-17c0-4f77-ba43-540f9524a8a9-kube-api-access-ztpvf\" (UniqueName: \"kubernetes.io/projected/dc3b282b-17c0-4f77-ba43-540f9524a8a9-kube-api-access-ztpvf\") pod \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") "
Oct 30 19:09:35.899846 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.899831    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-credential-socket\") pod \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\" (UID: \"dc3b282b-17c0-4f77-ba43-540f9524a8a9\") "
Oct 30 19:09:35.900469 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.900450    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:35.900558 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.900475    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:35.900558 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.900489    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c5b2946-7818-4067-976a-7e08e0f5c66c-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:35.900558 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.900505    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:35.900558 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.900518    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:35.900558 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.900532    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:35.900558 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.900547    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c5b2946-7818-4067-976a-7e08e0f5c66c-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:35.900856 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.900561    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w2v9\" (UniqueName: \"kubernetes.io/projected/4c5b2946-7818-4067-976a-7e08e0f5c66c-kube-api-access-6w2v9\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:35.900856 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.900575    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c5b2946-7818-4067-976a-7e08e0f5c66c-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:35.900856 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.900789    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-credential-socket" pod "dc3b282b-17c0-4f77-ba43-540f9524a8a9" (UID: "dc3b282b-17c0-4f77-ba43-540f9524a8a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:35.901000 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.900981    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-workload-socket" pod "dc3b282b-17c0-4f77-ba43-540f9524a8a9" (UID: "dc3b282b-17c0-4f77-ba43-540f9524a8a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:35.901302 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.901277    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istiod-ca-cert" pod "dc3b282b-17c0-4f77-ba43-540f9524a8a9" (UID: "dc3b282b-17c0-4f77-ba43-540f9524a8a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:35.902392 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.902368    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-data" pod "dc3b282b-17c0-4f77-ba43-540f9524a8a9" (UID: "dc3b282b-17c0-4f77-ba43-540f9524a8a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:35.902581 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.902554    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-workload-certs" pod "dc3b282b-17c0-4f77-ba43-540f9524a8a9" (UID: "dc3b282b-17c0-4f77-ba43-540f9524a8a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:35.903563 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.903541    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-token" pod "dc3b282b-17c0-4f77-ba43-540f9524a8a9" (UID: "dc3b282b-17c0-4f77-ba43-540f9524a8a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:35.906415 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.906365    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-envoy" pod "dc3b282b-17c0-4f77-ba43-540f9524a8a9" (UID: "dc3b282b-17c0-4f77-ba43-540f9524a8a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:35.906415 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.906392    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc3b282b-17c0-4f77-ba43-540f9524a8a9-kube-api-access-ztpvf" pod "dc3b282b-17c0-4f77-ba43-540f9524a8a9" (UID: "dc3b282b-17c0-4f77-ba43-540f9524a8a9"). InnerVolumeSpecName "kube-api-access-ztpvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:35.907447 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:35.907428    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-podinfo" pod "dc3b282b-17c0-4f77-ba43-540f9524a8a9" (UID: "dc3b282b-17c0-4f77-ba43-540f9524a8a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:09:36.001190 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.001157    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f53aa846-ec01-4b44-a518-49a558965906-kube-api-access-cwzxl\" (UniqueName: \"kubernetes.io/projected/f53aa846-ec01-4b44-a518-49a558965906-kube-api-access-cwzxl\") pod \"f53aa846-ec01-4b44-a518-49a558965906\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") "
Oct 30 19:09:36.001190 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.001195    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-token\" (UniqueName: \"kubernetes.io/projected/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-token\") pod \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") "
Oct 30 19:09:36.001415 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.001219    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-workload-certs\") pod \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") "
Oct 30 19:09:36.001415 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.001241    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f53aa846-ec01-4b44-a518-49a558965906-istio-token\" (UniqueName: \"kubernetes.io/projected/f53aa846-ec01-4b44-a518-49a558965906-istio-token\") pod \"f53aa846-ec01-4b44-a518-49a558965906\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") "
Oct 30 19:09:36.001415 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.001278    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-credential-socket\") pod \"f53aa846-ec01-4b44-a518-49a558965906\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") "
Oct 30 19:09:36.001415 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.001328    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f53aa846-ec01-4b44-a518-49a558965906-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53aa846-ec01-4b44-a518-49a558965906-istio-podinfo\") pod \"f53aa846-ec01-4b44-a518-49a558965906\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") "
Oct 30 19:09:36.001415 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.001384    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-workload-socket\") pod \"f53aa846-ec01-4b44-a518-49a558965906\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") "
Oct 30 19:09:36.001687 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.001423    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-workload-certs\") pod \"f53aa846-ec01-4b44-a518-49a558965906\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") "
Oct 30 19:09:36.001687 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.001444    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-data\") pod \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") "
Oct 30 19:09:36.001687 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.001491    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istiod-ca-cert\") pod \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") "
Oct 30 19:09:36.001687 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.001565    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-kube-api-access-d7hxw\" (UniqueName: \"kubernetes.io/projected/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-kube-api-access-d7hxw\") pod \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") "
Oct 30 19:09:36.001687 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.001579    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-workload-certs" pod "fa7b7f06-28aa-4a1f-ac1c-d1d363815699" (UID: "fa7b7f06-28aa-4a1f-ac1c-d1d363815699"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:36.001687 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.001594    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-credential-socket\") pod \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") "
Oct 30 19:09:36.001969 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.001719    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-podinfo\") pod \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") "
Oct 30 19:09:36.001969 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.001751    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-istio-data\") pod \"f53aa846-ec01-4b44-a518-49a558965906\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") "
Oct 30 19:09:36.001969 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.001778    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-data" pod "fa7b7f06-28aa-4a1f-ac1c-d1d363815699" (UID: "fa7b7f06-28aa-4a1f-ac1c-d1d363815699"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:36.001969 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.001803    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-envoy\") pod \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") "
Oct 30 19:09:36.001969 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.001896    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-istio-envoy\") pod \"f53aa846-ec01-4b44-a518-49a558965906\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") "
Oct 30 19:09:36.001969 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.001929    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-credential-socket" pod "fa7b7f06-28aa-4a1f-ac1c-d1d363815699" (UID: "fa7b7f06-28aa-4a1f-ac1c-d1d363815699"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:36.001969 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.001955    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istiod-ca-cert" pod "fa7b7f06-28aa-4a1f-ac1c-d1d363815699" (UID: "fa7b7f06-28aa-4a1f-ac1c-d1d363815699"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:36.002296 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.001999    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f53aa846-ec01-4b44-a518-49a558965906-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53aa846-ec01-4b44-a518-49a558965906-istiod-ca-cert\") pod \"f53aa846-ec01-4b44-a518-49a558965906\" (UID: \"f53aa846-ec01-4b44-a518-49a558965906\") "
Oct 30 19:09:36.002296 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.002028    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-workload-socket\") pod \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\" (UID: \"fa7b7f06-28aa-4a1f-ac1c-d1d363815699\") "
Oct 30 19:09:36.002405 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.002368    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-credential-socket" pod "f53aa846-ec01-4b44-a518-49a558965906" (UID: "f53aa846-ec01-4b44-a518-49a558965906"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:36.002714 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.002690    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-workload-certs" pod "f53aa846-ec01-4b44-a518-49a558965906" (UID: "f53aa846-ec01-4b44-a518-49a558965906"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:36.002823 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.002797    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-workload-socket" pod "f53aa846-ec01-4b44-a518-49a558965906" (UID: "f53aa846-ec01-4b44-a518-49a558965906"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:36.003550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.002978    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztpvf\" (UniqueName: \"kubernetes.io/projected/dc3b282b-17c0-4f77-ba43-540f9524a8a9-kube-api-access-ztpvf\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.003550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.002991    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-istio-data" pod "f53aa846-ec01-4b44-a518-49a558965906" (UID: "f53aa846-ec01-4b44-a518-49a558965906"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:36.003550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.003004    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.003550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.003020    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.003550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.003035    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.003550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.003050    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.003550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.003076    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.003550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.003090    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.003550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.003105    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.003550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.003121    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.003550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.003137    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.003550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.003151    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.003550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.003165    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.003550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.003180    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.003550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.003192    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc3b282b-17c0-4f77-ba43-540f9524a8a9-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.003550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.003207    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc3b282b-17c0-4f77-ba43-540f9524a8a9-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.003550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.003221    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.003550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.003237    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-workload-socket" pod "fa7b7f06-28aa-4a1f-ac1c-d1d363815699" (UID: "fa7b7f06-28aa-4a1f-ac1c-d1d363815699"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:36.003550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.003266    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f53aa846-ec01-4b44-a518-49a558965906-istiod-ca-cert" pod "f53aa846-ec01-4b44-a518-49a558965906" (UID: "f53aa846-ec01-4b44-a518-49a558965906"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:36.004862 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.003949    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53aa846-ec01-4b44-a518-49a558965906-kube-api-access-cwzxl" pod "f53aa846-ec01-4b44-a518-49a558965906" (UID: "f53aa846-ec01-4b44-a518-49a558965906"). InnerVolumeSpecName "kube-api-access-cwzxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:36.004862 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.004336    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53aa846-ec01-4b44-a518-49a558965906-istio-token" pod "f53aa846-ec01-4b44-a518-49a558965906" (UID: "f53aa846-ec01-4b44-a518-49a558965906"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:36.004862 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.004427    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-kube-api-access-d7hxw" pod "fa7b7f06-28aa-4a1f-ac1c-d1d363815699" (UID: "fa7b7f06-28aa-4a1f-ac1c-d1d363815699"). InnerVolumeSpecName "kube-api-access-d7hxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:36.004862 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.004471    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-token" pod "fa7b7f06-28aa-4a1f-ac1c-d1d363815699" (UID: "fa7b7f06-28aa-4a1f-ac1c-d1d363815699"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:36.005162 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.005130    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-envoy" pod "fa7b7f06-28aa-4a1f-ac1c-d1d363815699" (UID: "fa7b7f06-28aa-4a1f-ac1c-d1d363815699"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:36.005385 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.005360    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-podinfo" pod "fa7b7f06-28aa-4a1f-ac1c-d1d363815699" (UID: "fa7b7f06-28aa-4a1f-ac1c-d1d363815699"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:09:36.005480 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.005403    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f53aa846-ec01-4b44-a518-49a558965906-istio-podinfo" pod "f53aa846-ec01-4b44-a518-49a558965906" (UID: "f53aa846-ec01-4b44-a518-49a558965906"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:09:36.005527 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.005481    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-istio-envoy" pod "f53aa846-ec01-4b44-a518-49a558965906" (UID: "f53aa846-ec01-4b44-a518-49a558965906"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:09:36.055700 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.055669    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:36.104633 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.104596    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.104633 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.104632    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53aa846-ec01-4b44-a518-49a558965906-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.104852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.104648    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.104852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.104664    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwzxl\" (UniqueName: \"kubernetes.io/projected/f53aa846-ec01-4b44-a518-49a558965906-kube-api-access-cwzxl\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.104852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.104681    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.104852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.104695    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53aa846-ec01-4b44-a518-49a558965906-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.104852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.104708    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53aa846-ec01-4b44-a518-49a558965906-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.104852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.104721    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7hxw\" (UniqueName: \"kubernetes.io/projected/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-kube-api-access-d7hxw\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.104852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.104736    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.104852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.104749    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53aa846-ec01-4b44-a518-49a558965906-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.104852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.104762    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa7b7f06-28aa-4a1f-ac1c-d1d363815699-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:36.209752 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.209703    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"]
Oct 30 19:09:36.213899 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.213868    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-cpcj4"]
Oct 30 19:09:36.257314 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.257287    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:36.420603 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.420522    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"]
Oct 30 19:09:36.425009 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.424983    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gt964"]
Oct 30 19:09:36.837631 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.837541    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"
Oct 30 19:09:36.838520 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.838491    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c5b2946-7818-4067-976a-7e08e0f5c66c" path="/var/lib/kubelet/pods/4c5b2946-7818-4067-976a-7e08e0f5c66c/volumes"
Oct 30 19:09:36.839230 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.839208    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc3b282b-17c0-4f77-ba43-540f9524a8a9" path="/var/lib/kubelet/pods/dc3b282b-17c0-4f77-ba43-540f9524a8a9/volumes"
Oct 30 19:09:36.840118 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.840094    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g" event={"ID":"f53aa846-ec01-4b44-a518-49a558965906","Type":"ContainerDied","Data":"ceebc51234837bfe3a9bc708468043ad6c31b49abce9cc097ae962a12448e578"}
Oct 30 19:09:36.845380 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.845360    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"
Oct 30 19:09:36.845516 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.845418    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2" event={"ID":"fa7b7f06-28aa-4a1f-ac1c-d1d363815699","Type":"ContainerDied","Data":"f143be78fd8db90d978dcfaba8963d971ae760ed22e1921e4db78e2085342e1a"}
Oct 30 19:09:36.845516 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.845489    4924 scope.go:122] "RemoveContainer" containerID="30d947932b02122854a92acc189c210cd0e0bdcbf032836c9f5dd7fddcb3afca"
Oct 30 19:09:36.893806 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.893777    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv"]
Oct 30 19:09:36.894760 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.894739    4924 cpu_manager.go:414] "containerMap: removing container" podUID="dc3b282b-17c0-4f77-ba43-540f9524a8a9" containerName="istio-proxy"
Oct 30 19:09:36.894760 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.894759    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc3b282b-17c0-4f77-ba43-540f9524a8a9" containerName="istio-proxy"
Oct 30 19:09:36.894932 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.894781    4924 cpu_manager.go:414] "containerMap: removing container" podUID="f53aa846-ec01-4b44-a518-49a558965906" containerName="istio-proxy"
Oct 30 19:09:36.894932 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.894789    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f53aa846-ec01-4b44-a518-49a558965906" containerName="istio-proxy"
Oct 30 19:09:36.894932 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.894808    4924 cpu_manager.go:414] "containerMap: removing container" podUID="1706cc50-6d51-4e2a-8cf1-308f6e2187b0" containerName="discovery"
Oct 30 19:09:36.894932 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.894815    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1706cc50-6d51-4e2a-8cf1-308f6e2187b0" containerName="discovery"
Oct 30 19:09:36.894932 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.894839    4924 cpu_manager.go:414] "containerMap: removing container" podUID="34dfd0fe-b500-4df5-b191-6085a5d8dde5" containerName="istio-proxy"
Oct 30 19:09:36.894932 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.894846    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34dfd0fe-b500-4df5-b191-6085a5d8dde5" containerName="istio-proxy"
Oct 30 19:09:36.894932 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.894857    4924 cpu_manager.go:414] "containerMap: removing container" podUID="922dfc11-a4ba-4d50-a50f-b5155d277792" containerName="istio-proxy"
Oct 30 19:09:36.894932 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.894865    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="922dfc11-a4ba-4d50-a50f-b5155d277792" containerName="istio-proxy"
Oct 30 19:09:36.894932 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.894885    4924 cpu_manager.go:414] "containerMap: removing container" podUID="4c5b2946-7818-4067-976a-7e08e0f5c66c" containerName="istio-proxy"
Oct 30 19:09:36.894932 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.894891    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c5b2946-7818-4067-976a-7e08e0f5c66c" containerName="istio-proxy"
Oct 30 19:09:36.894932 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.894911    4924 cpu_manager.go:414] "containerMap: removing container" podUID="fa7b7f06-28aa-4a1f-ac1c-d1d363815699" containerName="istio-proxy"
Oct 30 19:09:36.894932 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.894918    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa7b7f06-28aa-4a1f-ac1c-d1d363815699" containerName="istio-proxy"
Oct 30 19:09:36.894932 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.894941    4924 cpu_manager.go:414] "containerMap: removing container" podUID="53b2edae-97f1-44e9-9580-29d605357c8f" containerName="istio-proxy"
Oct 30 19:09:36.895400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.894948    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b2edae-97f1-44e9-9580-29d605357c8f" containerName="istio-proxy"
Oct 30 19:09:36.895400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.894962    4924 cpu_manager.go:414] "containerMap: removing container" podUID="7d743094-1851-4c3a-9866-ecb1062c69b6" containerName="istio-proxy"
Oct 30 19:09:36.895400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.894968    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d743094-1851-4c3a-9866-ecb1062c69b6" containerName="istio-proxy"
Oct 30 19:09:36.895400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.894987    4924 cpu_manager.go:414] "containerMap: removing container" podUID="2eaee46c-23ef-410a-a5b6-450eba9711cc" containerName="istio-proxy"
Oct 30 19:09:36.895400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.894994    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eaee46c-23ef-410a-a5b6-450eba9711cc" containerName="istio-proxy"
Oct 30 19:09:36.895400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.895159    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc3b282b-17c0-4f77-ba43-540f9524a8a9" containerName="istio-proxy"
Oct 30 19:09:36.895400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.895171    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="f53aa846-ec01-4b44-a518-49a558965906" containerName="istio-proxy"
Oct 30 19:09:36.895400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.895181    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="1706cc50-6d51-4e2a-8cf1-308f6e2187b0" containerName="discovery"
Oct 30 19:09:36.895400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.895194    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="34dfd0fe-b500-4df5-b191-6085a5d8dde5" containerName="istio-proxy"
Oct 30 19:09:36.895400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.895201    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="922dfc11-a4ba-4d50-a50f-b5155d277792" containerName="istio-proxy"
Oct 30 19:09:36.895400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.895210    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c5b2946-7818-4067-976a-7e08e0f5c66c" containerName="istio-proxy"
Oct 30 19:09:36.895400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.895220    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa7b7f06-28aa-4a1f-ac1c-d1d363815699" containerName="istio-proxy"
Oct 30 19:09:36.895400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.895228    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="53b2edae-97f1-44e9-9580-29d605357c8f" containerName="istio-proxy"
Oct 30 19:09:36.895400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.895235    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d743094-1851-4c3a-9866-ecb1062c69b6" containerName="istio-proxy"
Oct 30 19:09:36.895400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.895241    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eaee46c-23ef-410a-a5b6-450eba9711cc" containerName="istio-proxy"
Oct 30 19:09:36.899818 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.899799    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv"
Oct 30 19:09:36.916439 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.916413    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv\" (UID: \"9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv"
Oct 30 19:09:36.916513 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.916447    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snd4m\" (UniqueName: \"kubernetes.io/projected/9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8-kube-api-access-snd4m\") pod \"maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv\" (UID: \"9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv"
Oct 30 19:09:36.916550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.916513    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv\" (UID: \"9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv"
Oct 30 19:09:36.949498 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.949438    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:09:36.949633 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.949529    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:09:36.949768 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.949631    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:09:36.949768 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.949662    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:09:36.951785 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.951755    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv"]
Oct 30 19:09:36.998214 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:36.998187    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"]
Oct 30 19:09:37.009656 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:37.009633    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-5lr2g"]
Oct 30 19:09:37.018756 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:37.018731    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv\" (UID: \"9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv"
Oct 30 19:09:37.018877 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:37.018764    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snd4m\" (UniqueName: \"kubernetes.io/projected/9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8-kube-api-access-snd4m\") pod \"maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv\" (UID: \"9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv"
Oct 30 19:09:37.018877 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:37.018814    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv\" (UID: \"9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv"
Oct 30 19:09:37.019486 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:37.019463    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv\" (UID: \"9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv"
Oct 30 19:09:37.021126 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:37.021106    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv\" (UID: \"9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv"
Oct 30 19:09:37.072831 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:37.072809    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snd4m\" (UniqueName: \"kubernetes.io/projected/9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8-kube-api-access-snd4m\") pod \"maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv\" (UID: \"9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv"
Oct 30 19:09:37.117143 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:37.116152    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"]
Oct 30 19:09:37.128437 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:37.128411    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-649d16-openshift-default-dbf76bc49-gtrr2"]
Oct 30 19:09:37.213323 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:37.213298    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv"
Oct 30 19:09:37.587683 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:37.587651    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv"]
Oct 30 19:09:37.587876 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:09:37.587841    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b3e536a_c40e_4fb9_8dde_c8e003c0b0d8.slice/crio-f8981b8bf8421f78659cdedc895a4eadb92ed8adaceda9a1e67646f998b129cc WatchSource:0}: Error finding container f8981b8bf8421f78659cdedc895a4eadb92ed8adaceda9a1e67646f998b129cc: Status 404 returned error can't find the container with id f8981b8bf8421f78659cdedc895a4eadb92ed8adaceda9a1e67646f998b129cc
Oct 30 19:09:37.858256 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:37.858224    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv" event={"ID":"9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8","Type":"ContainerStarted","Data":"f8981b8bf8421f78659cdedc895a4eadb92ed8adaceda9a1e67646f998b129cc"}
Oct 30 19:09:38.770926 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:38.770850    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:38.840610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:38.840341    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f53aa846-ec01-4b44-a518-49a558965906" path="/var/lib/kubelet/pods/f53aa846-ec01-4b44-a518-49a558965906/volumes"
Oct 30 19:09:38.843697 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:38.843635    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa7b7f06-28aa-4a1f-ac1c-d1d363815699" path="/var/lib/kubelet/pods/fa7b7f06-28aa-4a1f-ac1c-d1d363815699/volumes"
Oct 30 19:09:38.868616 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:38.868579    4924 generic.go:356] "Generic (PLEG): container finished" podID="9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8" containerID="83ce4836ff6d9b028d1de692a5dd50735756e3d75794d6aa1140a41dc52c9ffb" exitCode=0
Oct 30 19:09:38.868979 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:38.868662    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv" event={"ID":"9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8","Type":"ContainerDied","Data":"83ce4836ff6d9b028d1de692a5dd50735756e3d75794d6aa1140a41dc52c9ffb"}
Oct 30 19:09:38.953714 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:38.950905    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:39.328431 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:39.328401    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:39.527378 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:39.527343    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:40.030894 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.030864    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"]
Oct 30 19:09:40.031700 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.031682    4924 cpu_manager.go:414] "containerMap: removing container" podUID="f53aa846-ec01-4b44-a518-49a558965906" containerName="istio-proxy"
Oct 30 19:09:40.031758 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.031702    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f53aa846-ec01-4b44-a518-49a558965906" containerName="istio-proxy"
Oct 30 19:09:40.031810 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.031801    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="f53aa846-ec01-4b44-a518-49a558965906" containerName="istio-proxy"
Oct 30 19:09:40.036735 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.036712    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.073087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.072426    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-istio-envoy\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.073087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.072505    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34498bbd-c6eb-48b9-8003-d863dd955292-istio-podinfo\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.073087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.072562    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-workload-certs\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.073087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.072668    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-istio-data\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.073087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.072717    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34498bbd-c6eb-48b9-8003-d863dd955292-istiod-ca-cert\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.073087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.072745    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34498bbd-c6eb-48b9-8003-d863dd955292-istio-token\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.073087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.072803    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-credential-socket\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.073087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.072893    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-workload-socket\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.073087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.072960    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq7r7\" (UniqueName: \"kubernetes.io/projected/34498bbd-c6eb-48b9-8003-d863dd955292-kube-api-access-jq7r7\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.117085 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.116995    4924 scope.go:122] "RemoveContainer" containerID="b588edb5f3e457291b2e2577a73e2502023f492e2e21f98a9b020295a59c66f0"
Oct 30 19:09:40.120025 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.119925    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv"
Oct 30 19:09:40.122768 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.120733    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-8e31a7-openshift-default-dockercfg-xc59z\""
Oct 30 19:09:40.144385 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.144306    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"]
Oct 30 19:09:40.171477 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.171450    4924 scope.go:122] "RemoveContainer" containerID="b74a61d7149264b02e947fb5f4683ffe19cff93a963687cf8c58abf1c8fd61e5"
Oct 30 19:09:40.180684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.173584    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:40.180684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.177627    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8-maas-api-xks-ca\") pod \"9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8\" (UID: \"9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8\") "
Oct 30 19:09:40.180684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.177895    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8-maas-api-openshift-ca\") pod \"9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8\" (UID: \"9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8\") "
Oct 30 19:09:40.180684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.177936    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8-kube-api-access-snd4m\" (UniqueName: \"kubernetes.io/projected/9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8-kube-api-access-snd4m\") pod \"9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8\" (UID: \"9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8\") "
Oct 30 19:09:40.180684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.178250    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-workload-socket\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.180684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.178311    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq7r7\" (UniqueName: \"kubernetes.io/projected/34498bbd-c6eb-48b9-8003-d863dd955292-kube-api-access-jq7r7\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.180684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.178435    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-istio-envoy\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.180684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.178477    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34498bbd-c6eb-48b9-8003-d863dd955292-istio-podinfo\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.180684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.178507    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-workload-certs\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.180684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.178606    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-istio-data\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.180684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.178650    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34498bbd-c6eb-48b9-8003-d863dd955292-istiod-ca-cert\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.180684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.178677    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34498bbd-c6eb-48b9-8003-d863dd955292-istio-token\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.180684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.178728    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-credential-socket\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.180684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.179266    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-credential-socket\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.182558 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.182142    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-istio-data\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.185024 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.183674    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8-maas-api-openshift-ca" pod "9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8" (UID: "9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:09:40.185024 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.184004    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-workload-certs\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.185024 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.184418    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-workload-socket\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.185024 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.184982    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34498bbd-c6eb-48b9-8003-d863dd955292-istiod-ca-cert\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.200844 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.200521    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-istio-envoy\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.204278 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.201440    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34498bbd-c6eb-48b9-8003-d863dd955292-istio-podinfo\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.204278 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.202041    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8-kube-api-access-snd4m" pod "9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8" (UID: "9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8"). InnerVolumeSpecName "kube-api-access-snd4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:40.204671 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.204513    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8-maas-api-xks-ca" pod "9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8" (UID: "9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:09:40.239415 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.239386    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34498bbd-c6eb-48b9-8003-d863dd955292-istio-token\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.240267 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.240237    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq7r7\" (UniqueName: \"kubernetes.io/projected/34498bbd-c6eb-48b9-8003-d863dd955292-kube-api-access-jq7r7\") pod \"e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") " pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.244166 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.244143    4924 scope.go:122] "RemoveContainer" containerID="01686fbeec34af68c371f248740ee66d416de09aaa2b971e534dc0db12d1c8f9"
Oct 30 19:09:40.259543 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.259517    4924 scope.go:122] "RemoveContainer" containerID="7d53251a516c6b0f98b2602dc56c4bceb82481045ca48f283638f897f72df55f"
Oct 30 19:09:40.280250 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.280224    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:40.280351 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.280254    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:40.280351 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.280270    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snd4m\" (UniqueName: \"kubernetes.io/projected/9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8-kube-api-access-snd4m\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:40.495222 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.495186    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:40.799595 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.799428    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:40.887539 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.887513    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv"
Oct 30 19:09:40.887713 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.887519    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv" event={"ID":"9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8","Type":"ContainerDied","Data":"f8981b8bf8421f78659cdedc895a4eadb92ed8adaceda9a1e67646f998b129cc"}
Oct 30 19:09:40.887713 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.887628    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8981b8bf8421f78659cdedc895a4eadb92ed8adaceda9a1e67646f998b129cc"
Oct 30 19:09:40.974121 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:09:40.974054    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34498bbd_c6eb_48b9_8003_d863dd955292.slice/crio-fa07b98ddde96d6148650d455077c001891dbdb32c03a506eaea195ddaedc763 WatchSource:0}: Error finding container fa07b98ddde96d6148650d455077c001891dbdb32c03a506eaea195ddaedc763: Status 404 returned error can't find the container with id fa07b98ddde96d6148650d455077c001891dbdb32c03a506eaea195ddaedc763
Oct 30 19:09:40.974414 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.974390    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"]
Oct 30 19:09:40.976794 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.976764    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:09:40.976878 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.976832    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:09:40.976878 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:40.976862    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:09:41.541859 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:41.541826    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:41.897167 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:41.897132    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk" event={"ID":"34498bbd-c6eb-48b9-8003-d863dd955292","Type":"ContainerStarted","Data":"90b30198a6b8abc78e069f1c58737044cc9e9edc6e57e2517fd1e29f4a6a4921"}
Oct 30 19:09:41.897364 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:41.897191    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk" event={"ID":"34498bbd-c6eb-48b9-8003-d863dd955292","Type":"ContainerStarted","Data":"fa07b98ddde96d6148650d455077c001891dbdb32c03a506eaea195ddaedc763"}
Oct 30 19:09:42.102354 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:42.102306    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk" podStartSLOduration=3.102293159 podStartE2EDuration="3.102293159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:42.054501869 +0000 UTC m=+1983.331552315" watchObservedRunningTime="2026-10-30 19:09:42.102293159 +0000 UTC m=+1983.379343603"
Oct 30 19:09:42.496010 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:42.495973    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:42.501697 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:42.501673    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:42.905134 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:42.905105    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:42.906205 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:42.906184    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:09:43.953473 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:43.953394    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:44.858373 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:44.858339    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:45.131527 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:45.131437    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:45.712398 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:09:45.712364    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:09:48.052157 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:48.052119    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:48.575618 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:48.575585    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:48.755868 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:48.755824    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:48.931570 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:48.931533    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs"]
Oct 30 19:09:48.932244 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:48.932227    4924 cpu_manager.go:414] "containerMap: removing container" podUID="9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8" containerName="revoke-keys"
Oct 30 19:09:48.932326 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:48.932245    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8" containerName="revoke-keys"
Oct 30 19:09:48.932372 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:48.932363    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8" containerName="revoke-keys"
Oct 30 19:09:48.936242 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:48.936221    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs"
Oct 30 19:09:48.990622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:48.990592    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs"]
Oct 30 19:09:49.030651 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.030617    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kjtd\" (UniqueName: \"kubernetes.io/projected/fac381e9-2afe-4ea7-9b09-b35953682bec-kube-api-access-6kjtd\") pod \"payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs\" (UID: \"fac381e9-2afe-4ea7-9b09-b35953682bec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs"
Oct 30 19:09:49.030830 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.030660    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac381e9-2afe-4ea7-9b09-b35953682bec-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs\" (UID: \"fac381e9-2afe-4ea7-9b09-b35953682bec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs"
Oct 30 19:09:49.031580 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.031551    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8e31a7\""
Oct 30 19:09:49.083400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.083352    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd"]
Oct 30 19:09:49.088870 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.088842    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd"
Oct 30 19:09:49.132697 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.132660    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r6zt\" (UniqueName: \"kubernetes.io/projected/124af7f9-61f7-47d4-b932-796d501d85cb-kube-api-access-5r6zt\") pod \"payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd\" (UID: \"124af7f9-61f7-47d4-b932-796d501d85cb\") " pod="openshift-ingress/payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd"
Oct 30 19:09:49.133171 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.133142    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kjtd\" (UniqueName: \"kubernetes.io/projected/fac381e9-2afe-4ea7-9b09-b35953682bec-kube-api-access-6kjtd\") pod \"payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs\" (UID: \"fac381e9-2afe-4ea7-9b09-b35953682bec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs"
Oct 30 19:09:49.133329 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.133313    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac381e9-2afe-4ea7-9b09-b35953682bec-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs\" (UID: \"fac381e9-2afe-4ea7-9b09-b35953682bec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs"
Oct 30 19:09:49.133502 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.133482    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/124af7f9-61f7-47d4-b932-796d501d85cb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd\" (UID: \"124af7f9-61f7-47d4-b932-796d501d85cb\") " pod="openshift-ingress/payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd"
Oct 30 19:09:49.156087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.134551    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac381e9-2afe-4ea7-9b09-b35953682bec-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs\" (UID: \"fac381e9-2afe-4ea7-9b09-b35953682bec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs"
Oct 30 19:09:49.156087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.141383    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8e31a7-dockercfg-vkbqd\""
Oct 30 19:09:49.189131 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.187591    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd"]
Oct 30 19:09:49.189759 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.189699    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kjtd\" (UniqueName: \"kubernetes.io/projected/fac381e9-2afe-4ea7-9b09-b35953682bec-kube-api-access-6kjtd\") pod \"payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs\" (UID: \"fac381e9-2afe-4ea7-9b09-b35953682bec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs"
Oct 30 19:09:49.235623 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.235581    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/124af7f9-61f7-47d4-b932-796d501d85cb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd\" (UID: \"124af7f9-61f7-47d4-b932-796d501d85cb\") " pod="openshift-ingress/payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd"
Oct 30 19:09:49.235795 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.235760    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r6zt\" (UniqueName: \"kubernetes.io/projected/124af7f9-61f7-47d4-b932-796d501d85cb-kube-api-access-5r6zt\") pod \"payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd\" (UID: \"124af7f9-61f7-47d4-b932-796d501d85cb\") " pod="openshift-ingress/payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd"
Oct 30 19:09:49.237157 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.237135    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/124af7f9-61f7-47d4-b932-796d501d85cb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd\" (UID: \"124af7f9-61f7-47d4-b932-796d501d85cb\") " pod="openshift-ingress/payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd"
Oct 30 19:09:49.251650 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.251462    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs"
Oct 30 19:09:49.277179 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.277132    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:49.291776 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.291749    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r6zt\" (UniqueName: \"kubernetes.io/projected/124af7f9-61f7-47d4-b932-796d501d85cb-kube-api-access-5r6zt\") pod \"payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd\" (UID: \"124af7f9-61f7-47d4-b932-796d501d85cb\") " pod="openshift-ingress/payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd"
Oct 30 19:09:49.419171 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.419135    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd"
Oct 30 19:09:49.645181 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.645151    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs"]
Oct 30 19:09:49.645531 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:09:49.645506    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac381e9_2afe_4ea7_9b09_b35953682bec.slice/crio-0348c785ea40ca5c9e9a80f8a3daddfcc1b72112092b850117c36ccd9ab6a157 WatchSource:0}: Error finding container 0348c785ea40ca5c9e9a80f8a3daddfcc1b72112092b850117c36ccd9ab6a157: Status 404 returned error can't find the container with id 0348c785ea40ca5c9e9a80f8a3daddfcc1b72112092b850117c36ccd9ab6a157
Oct 30 19:09:49.801761 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.801735    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd"]
Oct 30 19:09:49.802302 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:09:49.802273    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124af7f9_61f7_47d4_b932_796d501d85cb.slice/crio-10dd8870a6a9a3e1ba3e4d8be72da166b0cd25e49264b4ad81d7757f78f9af44 WatchSource:0}: Error finding container 10dd8870a6a9a3e1ba3e4d8be72da166b0cd25e49264b4ad81d7757f78f9af44: Status 404 returned error can't find the container with id 10dd8870a6a9a3e1ba3e4d8be72da166b0cd25e49264b4ad81d7757f78f9af44
Oct 30 19:09:49.963863 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.963827    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd" event={"ID":"124af7f9-61f7-47d4-b932-796d501d85cb","Type":"ContainerStarted","Data":"92e8c8681abf62030001bce7dec7393e2fa099af0eb1672d7173bdf58aa6daba"}
Oct 30 19:09:49.964056 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.963893    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd" event={"ID":"124af7f9-61f7-47d4-b932-796d501d85cb","Type":"ContainerStarted","Data":"10dd8870a6a9a3e1ba3e4d8be72da166b0cd25e49264b4ad81d7757f78f9af44"}
Oct 30 19:09:49.964056 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.963927    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd"
Oct 30 19:09:49.965824 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.965790    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs" event={"ID":"fac381e9-2afe-4ea7-9b09-b35953682bec","Type":"ContainerStarted","Data":"0371d94615143ba5fc087a3ec18b443e29d34a4566c69e3a180516e8fdc14e5b"}
Oct 30 19:09:49.965939 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.965847    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs" event={"ID":"fac381e9-2afe-4ea7-9b09-b35953682bec","Type":"ContainerStarted","Data":"0348c785ea40ca5c9e9a80f8a3daddfcc1b72112092b850117c36ccd9ab6a157"}
Oct 30 19:09:49.965939 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:49.965880    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs"
Oct 30 19:09:50.076719 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:50.071755    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd" podStartSLOduration=1.071738431 podStartE2EDuration="1.071738431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:50.071300127 +0000 UTC m=+1991.348350573" watchObservedRunningTime="2026-10-30 19:09:50.071738431 +0000 UTC m=+1991.348788876"
Oct 30 19:09:50.076719 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:50.074754    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:50.181659 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:50.181602    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs" podStartSLOduration=2.181590051 podStartE2EDuration="2.181590051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:50.177987566 +0000 UTC m=+1991.455038010" watchObservedRunningTime="2026-10-30 19:09:50.181590051 +0000 UTC m=+1991.458640495"
Oct 30 19:09:50.320139 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:50.318802    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:54.653623 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:54.653582    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:09:59.747443 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:09:59.747400    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:00.985871 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:00.985837    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd"
Oct 30 19:10:00.986422 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:00.985922    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs"
Oct 30 19:10:03.105287 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:03.105254    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:03.874042 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:03.874007    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:04.382180 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.382148    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"]
Oct 30 19:10:04.386193 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.386172    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.440094 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.439186    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f3a0e7-openshift-default-dockercfg-nmtwr\""
Oct 30 19:10:04.467114 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.466035    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-credential-socket\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.467114 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.466168    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istiod-ca-cert\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.467114 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.466254    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-workload-socket\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.467114 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.466379    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mzrs\" (UniqueName: \"kubernetes.io/projected/0c9fa098-37e6-48ad-ba03-1879f9017f7c-kube-api-access-7mzrs\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.467114 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.466435    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-workload-certs\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.467114 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.466477    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-envoy\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.467114 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.466511    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-token\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.467114 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.466590    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-podinfo\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.467114 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.466631    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-data\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.554898 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.547648    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:04.561828 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.561801    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bb49af5-h8qkr"]
Oct 30 19:10:04.570305 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.570192    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-workload-certs\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.596277 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.571115    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-workload-certs\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.596563 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.573782    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb49af5-h8qkr"
Oct 30 19:10:04.597448 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.581557    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"]
Oct 30 19:10:04.598513 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.598481    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-envoy\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.598644 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.598548    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-token\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.599896 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.599870    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-podinfo\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.600021 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.599934    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-data\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.600139 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.600020    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-credential-socket\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.600215 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.600146    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istiod-ca-cert\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.600447 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.600251    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-workload-socket\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.600447 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.600432    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mzrs\" (UniqueName: \"kubernetes.io/projected/0c9fa098-37e6-48ad-ba03-1879f9017f7c-kube-api-access-7mzrs\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.601285 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.601256    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-workload-socket\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.601476 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.601449    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istiod-ca-cert\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.601791 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.601765    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-data\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.602184 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.602148    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-envoy\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.602364 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.602180    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-credential-socket\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.606115 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.605612    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-podinfo\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.620349 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.620321    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bb49af5-h8qkr"]
Oct 30 19:10:04.646898 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.646873    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:10:04.647316 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.647289    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:10:04.648834 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.648818    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:10:04.654818 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.654793    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-token\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.654904 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.654877    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mzrs\" (UniqueName: \"kubernetes.io/projected/0c9fa098-37e6-48ad-ba03-1879f9017f7c-kube-api-access-7mzrs\") pod \"e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") " pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.701498 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.701454    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:04.702177 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.702149    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn8fb\" (UniqueName: \"kubernetes.io/projected/2466b48e-af9c-4269-94f9-1e612e01a044-kube-api-access-wn8fb\") pod \"maas-api-revoke-sub-3bb49af5-h8qkr\" (UID: \"2466b48e-af9c-4269-94f9-1e612e01a044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb49af5-h8qkr"
Oct 30 19:10:04.702292 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.702268    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2466b48e-af9c-4269-94f9-1e612e01a044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bb49af5-h8qkr\" (UID: \"2466b48e-af9c-4269-94f9-1e612e01a044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb49af5-h8qkr"
Oct 30 19:10:04.702367 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.702353    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2466b48e-af9c-4269-94f9-1e612e01a044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bb49af5-h8qkr\" (UID: \"2466b48e-af9c-4269-94f9-1e612e01a044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb49af5-h8qkr"
Oct 30 19:10:04.746589 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.746551    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:10:04.771317 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.770929    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:04.806412 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.805799    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2466b48e-af9c-4269-94f9-1e612e01a044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bb49af5-h8qkr\" (UID: \"2466b48e-af9c-4269-94f9-1e612e01a044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb49af5-h8qkr"
Oct 30 19:10:04.806412 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.805923    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2466b48e-af9c-4269-94f9-1e612e01a044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bb49af5-h8qkr\" (UID: \"2466b48e-af9c-4269-94f9-1e612e01a044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb49af5-h8qkr"
Oct 30 19:10:04.809890 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.806942    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2466b48e-af9c-4269-94f9-1e612e01a044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bb49af5-h8qkr\" (UID: \"2466b48e-af9c-4269-94f9-1e612e01a044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb49af5-h8qkr"
Oct 30 19:10:04.809890 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.806987    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn8fb\" (UniqueName: \"kubernetes.io/projected/2466b48e-af9c-4269-94f9-1e612e01a044-kube-api-access-wn8fb\") pod \"maas-api-revoke-sub-3bb49af5-h8qkr\" (UID: \"2466b48e-af9c-4269-94f9-1e612e01a044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb49af5-h8qkr"
Oct 30 19:10:04.815447 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.815131    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2466b48e-af9c-4269-94f9-1e612e01a044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bb49af5-h8qkr\" (UID: \"2466b48e-af9c-4269-94f9-1e612e01a044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb49af5-h8qkr"
Oct 30 19:10:04.863124 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.862708    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn8fb\" (UniqueName: \"kubernetes.io/projected/2466b48e-af9c-4269-94f9-1e612e01a044-kube-api-access-wn8fb\") pod \"maas-api-revoke-sub-3bb49af5-h8qkr\" (UID: \"2466b48e-af9c-4269-94f9-1e612e01a044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb49af5-h8qkr"
Oct 30 19:10:04.954922 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:04.954819    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb49af5-h8qkr"
Oct 30 19:10:05.131319 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:10:05.131285    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9fa098_37e6_48ad_ba03_1879f9017f7c.slice/crio-a21e4af3ba878880be39a86ad637dc28d088f1257e908d7fde637cd3571db1f0 WatchSource:0}: Error finding container a21e4af3ba878880be39a86ad637dc28d088f1257e908d7fde637cd3571db1f0: Status 404 returned error can't find the container with id a21e4af3ba878880be39a86ad637dc28d088f1257e908d7fde637cd3571db1f0
Oct 30 19:10:05.131464 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:05.131344    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"]
Oct 30 19:10:05.134590 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:05.134558    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:10:05.134680 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:05.134624    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:10:05.134680 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:05.134654    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:10:05.329244 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:10:05.329201    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2466b48e_af9c_4269_94f9_1e612e01a044.slice/crio-d3237ed9fdf826d095201a392c4733e514ae0260c87ccd07db96f5af2e048668 WatchSource:0}: Error finding container d3237ed9fdf826d095201a392c4733e514ae0260c87ccd07db96f5af2e048668: Status 404 returned error can't find the container with id d3237ed9fdf826d095201a392c4733e514ae0260c87ccd07db96f5af2e048668
Oct 30 19:10:05.330324 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:05.330296    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bb49af5-h8qkr"]
Oct 30 19:10:05.669416 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:05.669377    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:05.913892 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:05.913861    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:06.092440 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:06.092352    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb49af5-h8qkr" event={"ID":"2466b48e-af9c-4269-94f9-1e612e01a044","Type":"ContainerStarted","Data":"44a90434f956799803869114fd1497046f18283a3e4bd433332686613b990c3b"}
Oct 30 19:10:06.092440 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:06.092403    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb49af5-h8qkr" event={"ID":"2466b48e-af9c-4269-94f9-1e612e01a044","Type":"ContainerStarted","Data":"d3237ed9fdf826d095201a392c4733e514ae0260c87ccd07db96f5af2e048668"}
Oct 30 19:10:06.094093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:06.094038    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf" event={"ID":"0c9fa098-37e6-48ad-ba03-1879f9017f7c","Type":"ContainerStarted","Data":"a2461b074d3d188c16e488a0dc09b2941276e4a70b34af07f3fd43c345633329"}
Oct 30 19:10:06.094093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:06.094091    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf" event={"ID":"0c9fa098-37e6-48ad-ba03-1879f9017f7c","Type":"ContainerStarted","Data":"a21e4af3ba878880be39a86ad637dc28d088f1257e908d7fde637cd3571db1f0"}
Oct 30 19:10:06.098662 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:06.098634    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:06.204125 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:06.204047    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb49af5-h8qkr" podStartSLOduration=2.204027594 podStartE2EDuration="2.204027594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:06.200925766 +0000 UTC m=+2007.477976211" watchObservedRunningTime="2026-10-30 19:10:06.204027594 +0000 UTC m=+2007.481078040"
Oct 30 19:10:06.312645 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:06.312583    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf" podStartSLOduration=2.312568677 podStartE2EDuration="2.312568677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:06.310527838 +0000 UTC m=+2007.587578283" watchObservedRunningTime="2026-10-30 19:10:06.312568677 +0000 UTC m=+2007.589619122"
Oct 30 19:10:06.702215 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:06.702181    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:06.708223 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:06.708198    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:07.102742 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:07.102705    4924 generic.go:356] "Generic (PLEG): container finished" podID="2466b48e-af9c-4269-94f9-1e612e01a044" containerID="44a90434f956799803869114fd1497046f18283a3e4bd433332686613b990c3b" exitCode=0
Oct 30 19:10:07.102952 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:07.102791    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb49af5-h8qkr" event={"ID":"2466b48e-af9c-4269-94f9-1e612e01a044","Type":"ContainerDied","Data":"44a90434f956799803869114fd1497046f18283a3e4bd433332686613b990c3b"}
Oct 30 19:10:07.103132 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:07.103101    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:07.104169 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:07.104151    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:10:08.279314 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:08.279289    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb49af5-h8qkr"
Oct 30 19:10:08.361438 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:08.361398    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2466b48e-af9c-4269-94f9-1e612e01a044-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2466b48e-af9c-4269-94f9-1e612e01a044-maas-api-xks-ca\") pod \"2466b48e-af9c-4269-94f9-1e612e01a044\" (UID: \"2466b48e-af9c-4269-94f9-1e612e01a044\") "
Oct 30 19:10:08.361659 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:08.361584    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2466b48e-af9c-4269-94f9-1e612e01a044-kube-api-access-wn8fb\" (UniqueName: \"kubernetes.io/projected/2466b48e-af9c-4269-94f9-1e612e01a044-kube-api-access-wn8fb\") pod \"2466b48e-af9c-4269-94f9-1e612e01a044\" (UID: \"2466b48e-af9c-4269-94f9-1e612e01a044\") "
Oct 30 19:10:08.361659 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:08.361629    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2466b48e-af9c-4269-94f9-1e612e01a044-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2466b48e-af9c-4269-94f9-1e612e01a044-maas-api-openshift-ca\") pod \"2466b48e-af9c-4269-94f9-1e612e01a044\" (UID: \"2466b48e-af9c-4269-94f9-1e612e01a044\") "
Oct 30 19:10:08.362082 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:08.362031    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2466b48e-af9c-4269-94f9-1e612e01a044-maas-api-openshift-ca" pod "2466b48e-af9c-4269-94f9-1e612e01a044" (UID: "2466b48e-af9c-4269-94f9-1e612e01a044"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:10:08.362335 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:08.362320    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2466b48e-af9c-4269-94f9-1e612e01a044-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:08.363539 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:08.363516    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2466b48e-af9c-4269-94f9-1e612e01a044-maas-api-xks-ca" pod "2466b48e-af9c-4269-94f9-1e612e01a044" (UID: "2466b48e-af9c-4269-94f9-1e612e01a044"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:10:08.363727 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:08.363707    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2466b48e-af9c-4269-94f9-1e612e01a044-kube-api-access-wn8fb" pod "2466b48e-af9c-4269-94f9-1e612e01a044" (UID: "2466b48e-af9c-4269-94f9-1e612e01a044"). InnerVolumeSpecName "kube-api-access-wn8fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:08.464133 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:08.464098    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2466b48e-af9c-4269-94f9-1e612e01a044-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:08.464133 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:08.464128    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn8fb\" (UniqueName: \"kubernetes.io/projected/2466b48e-af9c-4269-94f9-1e612e01a044-kube-api-access-wn8fb\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:09.119322 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:09.119292    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb49af5-h8qkr"
Oct 30 19:10:09.119485 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:09.119291    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb49af5-h8qkr" event={"ID":"2466b48e-af9c-4269-94f9-1e612e01a044","Type":"ContainerDied","Data":"d3237ed9fdf826d095201a392c4733e514ae0260c87ccd07db96f5af2e048668"}
Oct 30 19:10:09.119485 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:09.119409    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3237ed9fdf826d095201a392c4733e514ae0260c87ccd07db96f5af2e048668"
Oct 30 19:10:09.190013 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:09.189978    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:09.404034 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:09.403958    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:09.586559 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:09.586527    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:11.323208 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:11.323174    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:13.911823 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:13.911792    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:14.117412 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.117376    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:14.287180 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.287103    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x"]
Oct 30 19:10:14.287826 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.287806    4924 cpu_manager.go:414] "containerMap: removing container" podUID="2466b48e-af9c-4269-94f9-1e612e01a044" containerName="revoke-keys"
Oct 30 19:10:14.287826 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.287825    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2466b48e-af9c-4269-94f9-1e612e01a044" containerName="revoke-keys"
Oct 30 19:10:14.287992 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.287962    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="2466b48e-af9c-4269-94f9-1e612e01a044" containerName="revoke-keys"
Oct 30 19:10:14.291690 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.291666    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x"
Oct 30 19:10:14.345030 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.344999    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x"]
Oct 30 19:10:14.352357 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.352107    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6n9q\" (UniqueName: \"kubernetes.io/projected/0e8d5d76-7d10-4af1-8689-77dd76070856-kube-api-access-v6n9q\") pod \"payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x\" (UID: \"0e8d5d76-7d10-4af1-8689-77dd76070856\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x"
Oct 30 19:10:14.352357 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.352193    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8d5d76-7d10-4af1-8689-77dd76070856-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x\" (UID: \"0e8d5d76-7d10-4af1-8689-77dd76070856\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x"
Oct 30 19:10:14.386537 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.386500    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f3a0e7\""
Oct 30 19:10:14.386701 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.386671    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f3a0e7-dockercfg-4lnrp\""
Oct 30 19:10:14.459086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.455153    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6n9q\" (UniqueName: \"kubernetes.io/projected/0e8d5d76-7d10-4af1-8689-77dd76070856-kube-api-access-v6n9q\") pod \"payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x\" (UID: \"0e8d5d76-7d10-4af1-8689-77dd76070856\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x"
Oct 30 19:10:14.459086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.455244    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8d5d76-7d10-4af1-8689-77dd76070856-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x\" (UID: \"0e8d5d76-7d10-4af1-8689-77dd76070856\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x"
Oct 30 19:10:14.459086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.456605    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8d5d76-7d10-4af1-8689-77dd76070856-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x\" (UID: \"0e8d5d76-7d10-4af1-8689-77dd76070856\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x"
Oct 30 19:10:14.501093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.489308    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f"]
Oct 30 19:10:14.535553 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.535518    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6n9q\" (UniqueName: \"kubernetes.io/projected/0e8d5d76-7d10-4af1-8689-77dd76070856-kube-api-access-v6n9q\") pod \"payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x\" (UID: \"0e8d5d76-7d10-4af1-8689-77dd76070856\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x"
Oct 30 19:10:14.579466 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.579385    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f"
Oct 30 19:10:14.580109 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.579252    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f"]
Oct 30 19:10:14.608437 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.608412    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x"
Oct 30 19:10:14.658192 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.658162    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9326b21d-3f94-4d9c-99f5-93e96de020bf-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f\" (UID: \"9326b21d-3f94-4d9c-99f5-93e96de020bf\") " pod="openshift-ingress/payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f"
Oct 30 19:10:14.658342 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.658326    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw6xg\" (UniqueName: \"kubernetes.io/projected/9326b21d-3f94-4d9c-99f5-93e96de020bf-kube-api-access-jw6xg\") pod \"payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f\" (UID: \"9326b21d-3f94-4d9c-99f5-93e96de020bf\") " pod="openshift-ingress/payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f"
Oct 30 19:10:14.759283 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.759249    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw6xg\" (UniqueName: \"kubernetes.io/projected/9326b21d-3f94-4d9c-99f5-93e96de020bf-kube-api-access-jw6xg\") pod \"payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f\" (UID: \"9326b21d-3f94-4d9c-99f5-93e96de020bf\") " pod="openshift-ingress/payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f"
Oct 30 19:10:14.759475 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.759396    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9326b21d-3f94-4d9c-99f5-93e96de020bf-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f\" (UID: \"9326b21d-3f94-4d9c-99f5-93e96de020bf\") " pod="openshift-ingress/payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f"
Oct 30 19:10:14.760134 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.760112    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9326b21d-3f94-4d9c-99f5-93e96de020bf-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f\" (UID: \"9326b21d-3f94-4d9c-99f5-93e96de020bf\") " pod="openshift-ingress/payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f"
Oct 30 19:10:14.814109 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.814053    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw6xg\" (UniqueName: \"kubernetes.io/projected/9326b21d-3f94-4d9c-99f5-93e96de020bf-kube-api-access-jw6xg\") pod \"payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f\" (UID: \"9326b21d-3f94-4d9c-99f5-93e96de020bf\") " pod="openshift-ingress/payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f"
Oct 30 19:10:14.901406 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.901377    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f"
Oct 30 19:10:14.995851 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:14.995815    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x"]
Oct 30 19:10:15.007791 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:10:15.007762    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8d5d76_7d10_4af1_8689_77dd76070856.slice/crio-88c5c56a1f03ebd6e345274b290ddf0074a5bc934235da4ae630369310bd3b30 WatchSource:0}: Error finding container 88c5c56a1f03ebd6e345274b290ddf0074a5bc934235da4ae630369310bd3b30: Status 404 returned error can't find the container with id 88c5c56a1f03ebd6e345274b290ddf0074a5bc934235da4ae630369310bd3b30
Oct 30 19:10:15.165908 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:15.165825    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x" event={"ID":"0e8d5d76-7d10-4af1-8689-77dd76070856","Type":"ContainerStarted","Data":"2be0add783dbbcab1079ffc5bf984b5619a3ebb21434e38a630b44c55682e773"}
Oct 30 19:10:15.165908 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:15.165880    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x" event={"ID":"0e8d5d76-7d10-4af1-8689-77dd76070856","Type":"ContainerStarted","Data":"88c5c56a1f03ebd6e345274b290ddf0074a5bc934235da4ae630369310bd3b30"}
Oct 30 19:10:15.166175 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:15.166010    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x"
Oct 30 19:10:15.271135 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:15.271112    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f"]
Oct 30 19:10:15.271723 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:10:15.271697    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9326b21d_3f94_4d9c_99f5_93e96de020bf.slice/crio-212e7497dd1264957b1cff3003174db0e264574e8ee4ccb3a2a1c1ac312f2baf WatchSource:0}: Error finding container 212e7497dd1264957b1cff3003174db0e264574e8ee4ccb3a2a1c1ac312f2baf: Status 404 returned error can't find the container with id 212e7497dd1264957b1cff3003174db0e264574e8ee4ccb3a2a1c1ac312f2baf
Oct 30 19:10:15.275328 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:15.275289    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x" podStartSLOduration=1.2752769609999999 podStartE2EDuration="1.275276961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:15.273041752 +0000 UTC m=+2016.550092198" watchObservedRunningTime="2026-10-30 19:10:15.275276961 +0000 UTC m=+2016.552327407"
Oct 30 19:10:15.712961 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:10:15.712930    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:10:16.174938 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:16.174904    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f" event={"ID":"9326b21d-3f94-4d9c-99f5-93e96de020bf","Type":"ContainerStarted","Data":"aa6bdf60801d333c730aeedce620826bb0b842d2457a5830cab43b4aaa74970f"}
Oct 30 19:10:16.175386 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:16.174959    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f" event={"ID":"9326b21d-3f94-4d9c-99f5-93e96de020bf","Type":"ContainerStarted","Data":"212e7497dd1264957b1cff3003174db0e264574e8ee4ccb3a2a1c1ac312f2baf"}
Oct 30 19:10:16.175386 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:16.174986    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f"
Oct 30 19:10:16.283682 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:16.283634    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f" podStartSLOduration=2.283618548 podStartE2EDuration="2.283618548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:16.281519231 +0000 UTC m=+2017.558569677" watchObservedRunningTime="2026-10-30 19:10:16.283618548 +0000 UTC m=+2017.560668993"
Oct 30 19:10:16.774492 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:16.774382    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:20.646084 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:20.645989    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:21.702238 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:21.701502    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:26.178330 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:26.178299    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x"
Oct 30 19:10:26.563925 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:26.563838    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:27.188488 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:27.188457    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f"
Oct 30 19:10:29.247388 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:29.247351    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"]
Oct 30 19:10:29.247801 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:29.247581    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn" podUID="816f124c-a4bb-478e-b48a-c8b91a4113d2" containerName="istio-proxy" containerID="cri-o://8e47758cc8abe73a548256eb8bca57e4ad7e7be1e983349a6150169d6ddb2a3a" gracePeriod=30
Oct 30 19:10:29.294002 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:29.293966    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"]
Oct 30 19:10:29.294273 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:29.294250    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74" podUID="7af3a685-9810-41d3-9ba3-97e510ce2d96" containerName="istio-proxy" containerID="cri-o://02c0211a7c28830e71815e9ce3e0c7d4d8a8b453c08bdb2ce0eac3cad3a7c8ea" gracePeriod=30
Oct 30 19:10:31.502452 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:31.502407    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:33.591603 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:33.591566    4924 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:10:33.592192 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:33.591641    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74" podUID="7af3a685-9810-41d3-9ba3-97e510ce2d96" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:10:33.724570 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:33.724534    4924 patch_prober.go:29] interesting pod/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:10:33.734564 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:33.725230    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn" podUID="816f124c-a4bb-478e-b48a-c8b91a4113d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:10:34.580482 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.580449    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 19:10:34.611245 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.611211    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 19:10:34.660951 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.660917    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-envoy\") pod \"816f124c-a4bb-478e-b48a-c8b91a4113d2\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") "
Oct 30 19:10:34.661219 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.660994    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-workload-socket\") pod \"816f124c-a4bb-478e-b48a-c8b91a4113d2\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") "
Oct 30 19:10:34.661219 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.661088    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-workload-certs\") pod \"816f124c-a4bb-478e-b48a-c8b91a4113d2\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") "
Oct 30 19:10:34.661219 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.661122    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-credential-socket\") pod \"816f124c-a4bb-478e-b48a-c8b91a4113d2\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") "
Oct 30 19:10:34.661219 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.661182    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/816f124c-a4bb-478e-b48a-c8b91a4113d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/816f124c-a4bb-478e-b48a-c8b91a4113d2-istiod-ca-cert\") pod \"816f124c-a4bb-478e-b48a-c8b91a4113d2\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") "
Oct 30 19:10:34.661219 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.661215    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-token\" (UniqueName: \"kubernetes.io/projected/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-token\") pod \"816f124c-a4bb-478e-b48a-c8b91a4113d2\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") "
Oct 30 19:10:34.661532 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.661256    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/816f124c-a4bb-478e-b48a-c8b91a4113d2-kube-api-access-k27wb\" (UniqueName: \"kubernetes.io/projected/816f124c-a4bb-478e-b48a-c8b91a4113d2-kube-api-access-k27wb\") pod \"816f124c-a4bb-478e-b48a-c8b91a4113d2\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") "
Oct 30 19:10:34.661532 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.661270    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-workload-socket" pod "816f124c-a4bb-478e-b48a-c8b91a4113d2" (UID: "816f124c-a4bb-478e-b48a-c8b91a4113d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:34.661532 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.661382    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-data\") pod \"816f124c-a4bb-478e-b48a-c8b91a4113d2\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") "
Oct 30 19:10:34.661532 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.661442    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-workload-certs" pod "816f124c-a4bb-478e-b48a-c8b91a4113d2" (UID: "816f124c-a4bb-478e-b48a-c8b91a4113d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:34.661532 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.661462    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-credential-socket" pod "816f124c-a4bb-478e-b48a-c8b91a4113d2" (UID: "816f124c-a4bb-478e-b48a-c8b91a4113d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:34.661806 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.661625    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-podinfo\") pod \"816f124c-a4bb-478e-b48a-c8b91a4113d2\" (UID: \"816f124c-a4bb-478e-b48a-c8b91a4113d2\") "
Oct 30 19:10:34.661806 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.661724    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-data" pod "816f124c-a4bb-478e-b48a-c8b91a4113d2" (UID: "816f124c-a4bb-478e-b48a-c8b91a4113d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:34.661919 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.661878    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/816f124c-a4bb-478e-b48a-c8b91a4113d2-istiod-ca-cert" pod "816f124c-a4bb-478e-b48a-c8b91a4113d2" (UID: "816f124c-a4bb-478e-b48a-c8b91a4113d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:10:34.662454 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.662415    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.662454 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.662442    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.662600 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.662457    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.662600 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.662471    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/816f124c-a4bb-478e-b48a-c8b91a4113d2-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.662600 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.662484    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.663406 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.663385    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-envoy" pod "816f124c-a4bb-478e-b48a-c8b91a4113d2" (UID: "816f124c-a4bb-478e-b48a-c8b91a4113d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:34.663806 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.663788    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-token" pod "816f124c-a4bb-478e-b48a-c8b91a4113d2" (UID: "816f124c-a4bb-478e-b48a-c8b91a4113d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:34.663904 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.663888    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/816f124c-a4bb-478e-b48a-c8b91a4113d2-kube-api-access-k27wb" pod "816f124c-a4bb-478e-b48a-c8b91a4113d2" (UID: "816f124c-a4bb-478e-b48a-c8b91a4113d2"). InnerVolumeSpecName "kube-api-access-k27wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:34.664104 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.664053    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-podinfo" pod "816f124c-a4bb-478e-b48a-c8b91a4113d2" (UID: "816f124c-a4bb-478e-b48a-c8b91a4113d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:10:34.763046 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.763007    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-podinfo\") pod \"7af3a685-9810-41d3-9ba3-97e510ce2d96\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") "
Oct 30 19:10:34.763264 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.763058    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-token\" (UniqueName: \"kubernetes.io/projected/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-token\") pod \"7af3a685-9810-41d3-9ba3-97e510ce2d96\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") "
Oct 30 19:10:34.763264 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.763110    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-data\") pod \"7af3a685-9810-41d3-9ba3-97e510ce2d96\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") "
Oct 30 19:10:34.763264 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.763190    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-envoy\") pod \"7af3a685-9810-41d3-9ba3-97e510ce2d96\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") "
Oct 30 19:10:34.763264 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.763218    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-credential-socket\") pod \"7af3a685-9810-41d3-9ba3-97e510ce2d96\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") "
Oct 30 19:10:34.763566 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.763274    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7af3a685-9810-41d3-9ba3-97e510ce2d96-kube-api-access-8sh66\" (UniqueName: \"kubernetes.io/projected/7af3a685-9810-41d3-9ba3-97e510ce2d96-kube-api-access-8sh66\") pod \"7af3a685-9810-41d3-9ba3-97e510ce2d96\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") "
Oct 30 19:10:34.763566 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.763328    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7af3a685-9810-41d3-9ba3-97e510ce2d96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af3a685-9810-41d3-9ba3-97e510ce2d96-istiod-ca-cert\") pod \"7af3a685-9810-41d3-9ba3-97e510ce2d96\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") "
Oct 30 19:10:34.763566 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.763436    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-workload-certs\") pod \"7af3a685-9810-41d3-9ba3-97e510ce2d96\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") "
Oct 30 19:10:34.763566 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.763495    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-workload-socket\") pod \"7af3a685-9810-41d3-9ba3-97e510ce2d96\" (UID: \"7af3a685-9810-41d3-9ba3-97e510ce2d96\") "
Oct 30 19:10:34.764322 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.764111    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-credential-socket" pod "7af3a685-9810-41d3-9ba3-97e510ce2d96" (UID: "7af3a685-9810-41d3-9ba3-97e510ce2d96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:34.764322 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.764205    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af3a685-9810-41d3-9ba3-97e510ce2d96-istiod-ca-cert" pod "7af3a685-9810-41d3-9ba3-97e510ce2d96" (UID: "7af3a685-9810-41d3-9ba3-97e510ce2d96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:10:34.764507 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.764356    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-workload-socket" pod "7af3a685-9810-41d3-9ba3-97e510ce2d96" (UID: "7af3a685-9810-41d3-9ba3-97e510ce2d96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:34.764905 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.764557    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-workload-certs" pod "7af3a685-9810-41d3-9ba3-97e510ce2d96" (UID: "7af3a685-9810-41d3-9ba3-97e510ce2d96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:34.764905 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.764580    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.764905 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.764611    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.764905 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.764628    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/816f124c-a4bb-478e-b48a-c8b91a4113d2-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.764905 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.764645    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k27wb\" (UniqueName: \"kubernetes.io/projected/816f124c-a4bb-478e-b48a-c8b91a4113d2-kube-api-access-k27wb\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.764905 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.764661    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.764905 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.764744    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-data" pod "7af3a685-9810-41d3-9ba3-97e510ce2d96" (UID: "7af3a685-9810-41d3-9ba3-97e510ce2d96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:34.765666 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.765639    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-podinfo" pod "7af3a685-9810-41d3-9ba3-97e510ce2d96" (UID: "7af3a685-9810-41d3-9ba3-97e510ce2d96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:10:34.766761 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.766739    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-envoy" pod "7af3a685-9810-41d3-9ba3-97e510ce2d96" (UID: "7af3a685-9810-41d3-9ba3-97e510ce2d96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:10:34.766853 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.766791    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-token" pod "7af3a685-9810-41d3-9ba3-97e510ce2d96" (UID: "7af3a685-9810-41d3-9ba3-97e510ce2d96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:34.766906 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.766894    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af3a685-9810-41d3-9ba3-97e510ce2d96-kube-api-access-8sh66" pod "7af3a685-9810-41d3-9ba3-97e510ce2d96" (UID: "7af3a685-9810-41d3-9ba3-97e510ce2d96"). InnerVolumeSpecName "kube-api-access-8sh66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:34.866043 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.866008    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.866043 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.866043    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.866287 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.866057    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.866287 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.866114    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.866287 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.866127    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.866287 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.866143    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sh66\" (UniqueName: \"kubernetes.io/projected/7af3a685-9810-41d3-9ba3-97e510ce2d96-kube-api-access-8sh66\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.866287 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.866157    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af3a685-9810-41d3-9ba3-97e510ce2d96-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:34.866287 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:34.866170    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af3a685-9810-41d3-9ba3-97e510ce2d96-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:35.327034 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:35.326996    4924 generic.go:356] "Generic (PLEG): container finished" podID="7af3a685-9810-41d3-9ba3-97e510ce2d96" containerID="02c0211a7c28830e71815e9ce3e0c7d4d8a8b453c08bdb2ce0eac3cad3a7c8ea" exitCode=0
Oct 30 19:10:35.327205 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:35.327051    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"
Oct 30 19:10:35.327205 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:35.327091    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74" event={"ID":"7af3a685-9810-41d3-9ba3-97e510ce2d96","Type":"ContainerDied","Data":"02c0211a7c28830e71815e9ce3e0c7d4d8a8b453c08bdb2ce0eac3cad3a7c8ea"}
Oct 30 19:10:35.327205 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:35.327155    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74" event={"ID":"7af3a685-9810-41d3-9ba3-97e510ce2d96","Type":"ContainerDied","Data":"606693a8796bc424dd3042dbb62af1a4f678409baf28f656488134ae98fdde6a"}
Oct 30 19:10:35.327205 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:35.327185    4924 scope.go:122] "RemoveContainer" containerID="02c0211a7c28830e71815e9ce3e0c7d4d8a8b453c08bdb2ce0eac3cad3a7c8ea"
Oct 30 19:10:35.328697 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:35.328673    4924 generic.go:356] "Generic (PLEG): container finished" podID="816f124c-a4bb-478e-b48a-c8b91a4113d2" containerID="8e47758cc8abe73a548256eb8bca57e4ad7e7be1e983349a6150169d6ddb2a3a" exitCode=0
Oct 30 19:10:35.328822 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:35.328698    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn" event={"ID":"816f124c-a4bb-478e-b48a-c8b91a4113d2","Type":"ContainerDied","Data":"8e47758cc8abe73a548256eb8bca57e4ad7e7be1e983349a6150169d6ddb2a3a"}
Oct 30 19:10:35.328822 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:35.328729    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"
Oct 30 19:10:35.328822 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:35.328743    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn" event={"ID":"816f124c-a4bb-478e-b48a-c8b91a4113d2","Type":"ContainerDied","Data":"033debdce115fe8da3135b8a72ad22215736a208482481be11bdaa40a777b195"}
Oct 30 19:10:35.338007 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:35.337989    4924 scope.go:122] "RemoveContainer" containerID="02c0211a7c28830e71815e9ce3e0c7d4d8a8b453c08bdb2ce0eac3cad3a7c8ea"
Oct 30 19:10:35.338298 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:10:35.338277    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02c0211a7c28830e71815e9ce3e0c7d4d8a8b453c08bdb2ce0eac3cad3a7c8ea\": container with ID starting with 02c0211a7c28830e71815e9ce3e0c7d4d8a8b453c08bdb2ce0eac3cad3a7c8ea not found: ID does not exist" containerID="02c0211a7c28830e71815e9ce3e0c7d4d8a8b453c08bdb2ce0eac3cad3a7c8ea"
Oct 30 19:10:35.338382 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:35.338309    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c0211a7c28830e71815e9ce3e0c7d4d8a8b453c08bdb2ce0eac3cad3a7c8ea"} err="failed to get container status \"02c0211a7c28830e71815e9ce3e0c7d4d8a8b453c08bdb2ce0eac3cad3a7c8ea\": rpc error: code = NotFound desc = could not find container \"02c0211a7c28830e71815e9ce3e0c7d4d8a8b453c08bdb2ce0eac3cad3a7c8ea\": container with ID starting with 02c0211a7c28830e71815e9ce3e0c7d4d8a8b453c08bdb2ce0eac3cad3a7c8ea not found: ID does not exist"
Oct 30 19:10:35.338382 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:35.338331    4924 scope.go:122] "RemoveContainer" containerID="8e47758cc8abe73a548256eb8bca57e4ad7e7be1e983349a6150169d6ddb2a3a"
Oct 30 19:10:35.348188 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:35.348172    4924 scope.go:122] "RemoveContainer" containerID="8e47758cc8abe73a548256eb8bca57e4ad7e7be1e983349a6150169d6ddb2a3a"
Oct 30 19:10:35.348432 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:10:35.348416    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e47758cc8abe73a548256eb8bca57e4ad7e7be1e983349a6150169d6ddb2a3a\": container with ID starting with 8e47758cc8abe73a548256eb8bca57e4ad7e7be1e983349a6150169d6ddb2a3a not found: ID does not exist" containerID="8e47758cc8abe73a548256eb8bca57e4ad7e7be1e983349a6150169d6ddb2a3a"
Oct 30 19:10:35.348483 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:35.348439    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e47758cc8abe73a548256eb8bca57e4ad7e7be1e983349a6150169d6ddb2a3a"} err="failed to get container status \"8e47758cc8abe73a548256eb8bca57e4ad7e7be1e983349a6150169d6ddb2a3a\": rpc error: code = NotFound desc = could not find container \"8e47758cc8abe73a548256eb8bca57e4ad7e7be1e983349a6150169d6ddb2a3a\": container with ID starting with 8e47758cc8abe73a548256eb8bca57e4ad7e7be1e983349a6150169d6ddb2a3a not found: ID does not exist"
Oct 30 19:10:35.484876 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:35.484839    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"]
Oct 30 19:10:35.531661 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:35.531584    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-ghcfn"]
Oct 30 19:10:35.597095 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:35.597052    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"]
Oct 30 19:10:35.603795 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:35.603765    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-6fh74"]
Oct 30 19:10:36.362384 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:36.360244    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:36.836835 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:36.836751    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7af3a685-9810-41d3-9ba3-97e510ce2d96" path="/var/lib/kubelet/pods/7af3a685-9810-41d3-9ba3-97e510ce2d96/volumes"
Oct 30 19:10:36.837293 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:36.837278    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="816f124c-a4bb-478e-b48a-c8b91a4113d2" path="/var/lib/kubelet/pods/816f124c-a4bb-478e-b48a-c8b91a4113d2/volumes"
Oct 30 19:10:39.634709 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:39.634674    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:40.407142 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:40.407110    4924 scope.go:122] "RemoveContainer" containerID="ada40b7f7abe844ddb41950a27d93d871473b4ed9aac378e4cc8cb1730245b9a"
Oct 30 19:10:40.420354 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:40.420335    4924 scope.go:122] "RemoveContainer" containerID="d3c8055238d6e4012b85b8c35fc3575ce8b9436b199007ea94cca9e42868faa9"
Oct 30 19:10:40.430260 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:40.430238    4924 scope.go:122] "RemoveContainer" containerID="ab08e90d3071dded6a06178abfdb5c2d8a262973175b48f3aee4cf4eb8bc2c9a"
Oct 30 19:10:41.278406 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:41.278368    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:45.712609 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:10:45.712572    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:10:46.515511 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:46.515476    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:50.092712 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:50.092678    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8416e0-r2sv6"]
Oct 30 19:10:50.139300 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:50.139270    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed8416e0-r2sv6"]
Oct 30 19:10:50.837154 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:50.837124    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9cce67a-d961-4ba6-92d9-df9b2bea4919" path="/var/lib/kubelet/pods/a9cce67a-d961-4ba6-92d9-df9b2bea4919/volumes"
Oct 30 19:10:51.349346 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:51.349313    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cfb433-nr52q"]
Oct 30 19:10:51.349988 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:51.349967    4924 cpu_manager.go:414] "containerMap: removing container" podUID="7af3a685-9810-41d3-9ba3-97e510ce2d96" containerName="istio-proxy"
Oct 30 19:10:51.349988 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:51.349984    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7af3a685-9810-41d3-9ba3-97e510ce2d96" containerName="istio-proxy"
Oct 30 19:10:51.350213 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:51.350048    4924 cpu_manager.go:414] "containerMap: removing container" podUID="816f124c-a4bb-478e-b48a-c8b91a4113d2" containerName="istio-proxy"
Oct 30 19:10:51.350213 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:51.350058    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="816f124c-a4bb-478e-b48a-c8b91a4113d2" containerName="istio-proxy"
Oct 30 19:10:51.350347 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:51.350261    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="816f124c-a4bb-478e-b48a-c8b91a4113d2" containerName="istio-proxy"
Oct 30 19:10:51.350347 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:51.350280    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="7af3a685-9810-41d3-9ba3-97e510ce2d96" containerName="istio-proxy"
Oct 30 19:10:51.356186 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:51.356169    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cfb433-nr52q"
Oct 30 19:10:51.411652 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:51.411621    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cfb433-nr52q"]
Oct 30 19:10:51.452870 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:51.452840    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:10:51.453025 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:51.452899    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:10:51.454205 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:51.454183    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:10:51.454414 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:51.454397    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:10:51.516373 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:51.516349    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/597ba804-ba56-4aa4-9f94-a95506856b98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cfb433-nr52q\" (UID: \"597ba804-ba56-4aa4-9f94-a95506856b98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cfb433-nr52q"
Oct 30 19:10:51.516504 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:51.516411    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/597ba804-ba56-4aa4-9f94-a95506856b98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cfb433-nr52q\" (UID: \"597ba804-ba56-4aa4-9f94-a95506856b98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cfb433-nr52q"
Oct 30 19:10:51.516593 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:51.516569    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fswxs\" (UniqueName: \"kubernetes.io/projected/597ba804-ba56-4aa4-9f94-a95506856b98-kube-api-access-fswxs\") pod \"maas-api-revoke-keys-e2e-api-a-cfb433-nr52q\" (UID: \"597ba804-ba56-4aa4-9f94-a95506856b98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cfb433-nr52q"
Oct 30 19:10:51.617610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:51.617509    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/597ba804-ba56-4aa4-9f94-a95506856b98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cfb433-nr52q\" (UID: \"597ba804-ba56-4aa4-9f94-a95506856b98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cfb433-nr52q"
Oct 30 19:10:51.617778 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:51.617620    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/597ba804-ba56-4aa4-9f94-a95506856b98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cfb433-nr52q\" (UID: \"597ba804-ba56-4aa4-9f94-a95506856b98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cfb433-nr52q"
Oct 30 19:10:51.617778 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:51.617757    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fswxs\" (UniqueName: \"kubernetes.io/projected/597ba804-ba56-4aa4-9f94-a95506856b98-kube-api-access-fswxs\") pod \"maas-api-revoke-keys-e2e-api-a-cfb433-nr52q\" (UID: \"597ba804-ba56-4aa4-9f94-a95506856b98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cfb433-nr52q"
Oct 30 19:10:51.618319 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:51.618296    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/597ba804-ba56-4aa4-9f94-a95506856b98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cfb433-nr52q\" (UID: \"597ba804-ba56-4aa4-9f94-a95506856b98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cfb433-nr52q"
Oct 30 19:10:51.619921 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:51.619901    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/597ba804-ba56-4aa4-9f94-a95506856b98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cfb433-nr52q\" (UID: \"597ba804-ba56-4aa4-9f94-a95506856b98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cfb433-nr52q"
Oct 30 19:10:51.674024 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:51.674002    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fswxs\" (UniqueName: \"kubernetes.io/projected/597ba804-ba56-4aa4-9f94-a95506856b98-kube-api-access-fswxs\") pod \"maas-api-revoke-keys-e2e-api-a-cfb433-nr52q\" (UID: \"597ba804-ba56-4aa4-9f94-a95506856b98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cfb433-nr52q"
Oct 30 19:10:51.914114 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:51.914012    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:51.970400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:51.970366    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cfb433-nr52q"
Oct 30 19:10:52.339956 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:52.339893    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cfb433-nr52q"]
Oct 30 19:10:52.341247 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:10:52.341219    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod597ba804_ba56_4aa4_9f94_a95506856b98.slice/crio-bafa7cbf425abc2cd3608761a5c88cbbc9c80ae18378a731e18247740c8a2d7b WatchSource:0}: Error finding container bafa7cbf425abc2cd3608761a5c88cbbc9c80ae18378a731e18247740c8a2d7b: Status 404 returned error can't find the container with id bafa7cbf425abc2cd3608761a5c88cbbc9c80ae18378a731e18247740c8a2d7b
Oct 30 19:10:52.455212 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:52.455181    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cfb433-nr52q" event={"ID":"597ba804-ba56-4aa4-9f94-a95506856b98","Type":"ContainerStarted","Data":"bafa7cbf425abc2cd3608761a5c88cbbc9c80ae18378a731e18247740c8a2d7b"}
Oct 30 19:10:53.464311 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:53.464231    4924 generic.go:356] "Generic (PLEG): container finished" podID="597ba804-ba56-4aa4-9f94-a95506856b98" containerID="827a910a1ed81918a751825f88897fe449089ab857e688010a9c799607504fd8" exitCode=0
Oct 30 19:10:53.464677 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:53.464302    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cfb433-nr52q" event={"ID":"597ba804-ba56-4aa4-9f94-a95506856b98","Type":"ContainerDied","Data":"827a910a1ed81918a751825f88897fe449089ab857e688010a9c799607504fd8"}
Oct 30 19:10:54.638527 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:54.638501    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cfb433-nr52q"
Oct 30 19:10:54.767284 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:54.767190    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/597ba804-ba56-4aa4-9f94-a95506856b98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/597ba804-ba56-4aa4-9f94-a95506856b98-maas-api-openshift-ca\") pod \"597ba804-ba56-4aa4-9f94-a95506856b98\" (UID: \"597ba804-ba56-4aa4-9f94-a95506856b98\") "
Oct 30 19:10:54.767453 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:54.767298    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/597ba804-ba56-4aa4-9f94-a95506856b98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/597ba804-ba56-4aa4-9f94-a95506856b98-maas-api-xks-ca\") pod \"597ba804-ba56-4aa4-9f94-a95506856b98\" (UID: \"597ba804-ba56-4aa4-9f94-a95506856b98\") "
Oct 30 19:10:54.767453 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:54.767436    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/597ba804-ba56-4aa4-9f94-a95506856b98-kube-api-access-fswxs\" (UniqueName: \"kubernetes.io/projected/597ba804-ba56-4aa4-9f94-a95506856b98-kube-api-access-fswxs\") pod \"597ba804-ba56-4aa4-9f94-a95506856b98\" (UID: \"597ba804-ba56-4aa4-9f94-a95506856b98\") "
Oct 30 19:10:54.767770 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:54.767736    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/597ba804-ba56-4aa4-9f94-a95506856b98-maas-api-openshift-ca" pod "597ba804-ba56-4aa4-9f94-a95506856b98" (UID: "597ba804-ba56-4aa4-9f94-a95506856b98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:10:54.768416 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:54.768372    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/597ba804-ba56-4aa4-9f94-a95506856b98-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:54.771053 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:54.771024    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/597ba804-ba56-4aa4-9f94-a95506856b98-maas-api-xks-ca" pod "597ba804-ba56-4aa4-9f94-a95506856b98" (UID: "597ba804-ba56-4aa4-9f94-a95506856b98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:10:54.772466 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:54.772444    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/597ba804-ba56-4aa4-9f94-a95506856b98-kube-api-access-fswxs" pod "597ba804-ba56-4aa4-9f94-a95506856b98" (UID: "597ba804-ba56-4aa4-9f94-a95506856b98"). InnerVolumeSpecName "kube-api-access-fswxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:54.869301 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:54.869273    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fswxs\" (UniqueName: \"kubernetes.io/projected/597ba804-ba56-4aa4-9f94-a95506856b98-kube-api-access-fswxs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:54.869301 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:54.869300    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/597ba804-ba56-4aa4-9f94-a95506856b98-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:55.480625 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:55.480593    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cfb433-nr52q"
Oct 30 19:10:55.480814 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:55.480620    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cfb433-nr52q" event={"ID":"597ba804-ba56-4aa4-9f94-a95506856b98","Type":"ContainerDied","Data":"bafa7cbf425abc2cd3608761a5c88cbbc9c80ae18378a731e18247740c8a2d7b"}
Oct 30 19:10:55.480814 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:55.480673    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bafa7cbf425abc2cd3608761a5c88cbbc9c80ae18378a731e18247740c8a2d7b"
Oct 30 19:10:56.386086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:56.383013    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm"]
Oct 30 19:10:56.386086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:56.384162    4924 cpu_manager.go:414] "containerMap: removing container" podUID="597ba804-ba56-4aa4-9f94-a95506856b98" containerName="revoke-keys"
Oct 30 19:10:56.386086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:56.384190    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="597ba804-ba56-4aa4-9f94-a95506856b98" containerName="revoke-keys"
Oct 30 19:10:56.386086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:56.384402    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="597ba804-ba56-4aa4-9f94-a95506856b98" containerName="revoke-keys"
Oct 30 19:10:56.411423 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:56.411389    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm"
Oct 30 19:10:56.435610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:56.435571    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm"]
Oct 30 19:10:56.473401 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:56.473372    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:10:56.475423 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:56.475401    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:10:56.475423 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:56.475412    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:10:56.506949 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:56.506922    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82423fa5-f43a-4c81-ac33-df275d1bd095-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm\" (UID: \"82423fa5-f43a-4c81-ac33-df275d1bd095\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm"
Oct 30 19:10:56.507101 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:56.506975    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82423fa5-f43a-4c81-ac33-df275d1bd095-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm\" (UID: \"82423fa5-f43a-4c81-ac33-df275d1bd095\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm"
Oct 30 19:10:56.507195 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:56.507175    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtbvm\" (UniqueName: \"kubernetes.io/projected/82423fa5-f43a-4c81-ac33-df275d1bd095-kube-api-access-mtbvm\") pod \"maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm\" (UID: \"82423fa5-f43a-4c81-ac33-df275d1bd095\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm"
Oct 30 19:10:56.507657 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:56.507642    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:10:56.608152 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:56.608116    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtbvm\" (UniqueName: \"kubernetes.io/projected/82423fa5-f43a-4c81-ac33-df275d1bd095-kube-api-access-mtbvm\") pod \"maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm\" (UID: \"82423fa5-f43a-4c81-ac33-df275d1bd095\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm"
Oct 30 19:10:56.608314 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:56.608284    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82423fa5-f43a-4c81-ac33-df275d1bd095-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm\" (UID: \"82423fa5-f43a-4c81-ac33-df275d1bd095\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm"
Oct 30 19:10:56.608367 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:56.608352    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82423fa5-f43a-4c81-ac33-df275d1bd095-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm\" (UID: \"82423fa5-f43a-4c81-ac33-df275d1bd095\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm"
Oct 30 19:10:56.609012 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:56.608988    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82423fa5-f43a-4c81-ac33-df275d1bd095-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm\" (UID: \"82423fa5-f43a-4c81-ac33-df275d1bd095\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm"
Oct 30 19:10:56.610698 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:56.610672    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82423fa5-f43a-4c81-ac33-df275d1bd095-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm\" (UID: \"82423fa5-f43a-4c81-ac33-df275d1bd095\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm"
Oct 30 19:10:56.661870 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:56.661811    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtbvm\" (UniqueName: \"kubernetes.io/projected/82423fa5-f43a-4c81-ac33-df275d1bd095-kube-api-access-mtbvm\") pod \"maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm\" (UID: \"82423fa5-f43a-4c81-ac33-df275d1bd095\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm"
Oct 30 19:10:56.753382 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:56.753352    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm"
Oct 30 19:10:57.127274 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:57.127242    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm"]
Oct 30 19:10:57.128152 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:10:57.128121    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82423fa5_f43a_4c81_ac33_df275d1bd095.slice/crio-a98379bfa2363e3ab68a70095c5892fe1d8a49e084e2752c4710e047babe0b78 WatchSource:0}: Error finding container a98379bfa2363e3ab68a70095c5892fe1d8a49e084e2752c4710e047babe0b78: Status 404 returned error can't find the container with id a98379bfa2363e3ab68a70095c5892fe1d8a49e084e2752c4710e047babe0b78
Oct 30 19:10:57.176986 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:57.176948    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:10:57.498468 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:57.498429    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm" event={"ID":"82423fa5-f43a-4c81-ac33-df275d1bd095","Type":"ContainerStarted","Data":"a98379bfa2363e3ab68a70095c5892fe1d8a49e084e2752c4710e047babe0b78"}
Oct 30 19:10:58.508524 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:58.508436    4924 generic.go:356] "Generic (PLEG): container finished" podID="82423fa5-f43a-4c81-ac33-df275d1bd095" containerID="73373dfe5a4c261911414ffc16987c4260144075f27dc16229e4f2eff2c9301e" exitCode=0
Oct 30 19:10:58.508524 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:58.508489    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm" event={"ID":"82423fa5-f43a-4c81-ac33-df275d1bd095","Type":"ContainerDied","Data":"73373dfe5a4c261911414ffc16987c4260144075f27dc16229e4f2eff2c9301e"}
Oct 30 19:10:59.686747 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:59.686719    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm"
Oct 30 19:10:59.752530 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:59.752502    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82423fa5-f43a-4c81-ac33-df275d1bd095-kube-api-access-mtbvm\" (UniqueName: \"kubernetes.io/projected/82423fa5-f43a-4c81-ac33-df275d1bd095-kube-api-access-mtbvm\") pod \"82423fa5-f43a-4c81-ac33-df275d1bd095\" (UID: \"82423fa5-f43a-4c81-ac33-df275d1bd095\") "
Oct 30 19:10:59.752674 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:59.752656    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82423fa5-f43a-4c81-ac33-df275d1bd095-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82423fa5-f43a-4c81-ac33-df275d1bd095-maas-api-xks-ca\") pod \"82423fa5-f43a-4c81-ac33-df275d1bd095\" (UID: \"82423fa5-f43a-4c81-ac33-df275d1bd095\") "
Oct 30 19:10:59.752752 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:59.752703    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82423fa5-f43a-4c81-ac33-df275d1bd095-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82423fa5-f43a-4c81-ac33-df275d1bd095-maas-api-openshift-ca\") pod \"82423fa5-f43a-4c81-ac33-df275d1bd095\" (UID: \"82423fa5-f43a-4c81-ac33-df275d1bd095\") "
Oct 30 19:10:59.753119 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:59.753093    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82423fa5-f43a-4c81-ac33-df275d1bd095-maas-api-openshift-ca" pod "82423fa5-f43a-4c81-ac33-df275d1bd095" (UID: "82423fa5-f43a-4c81-ac33-df275d1bd095"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:10:59.753436 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:59.753416    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82423fa5-f43a-4c81-ac33-df275d1bd095-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:59.754662 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:59.754641    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82423fa5-f43a-4c81-ac33-df275d1bd095-maas-api-xks-ca" pod "82423fa5-f43a-4c81-ac33-df275d1bd095" (UID: "82423fa5-f43a-4c81-ac33-df275d1bd095"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:10:59.754756 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:59.754741    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82423fa5-f43a-4c81-ac33-df275d1bd095-kube-api-access-mtbvm" pod "82423fa5-f43a-4c81-ac33-df275d1bd095" (UID: "82423fa5-f43a-4c81-ac33-df275d1bd095"). InnerVolumeSpecName "kube-api-access-mtbvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:10:59.854606 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:59.854570    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtbvm\" (UniqueName: \"kubernetes.io/projected/82423fa5-f43a-4c81-ac33-df275d1bd095-kube-api-access-mtbvm\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:10:59.854606 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:10:59.854603    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82423fa5-f43a-4c81-ac33-df275d1bd095-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:00.522967 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:00.522931    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm"
Oct 30 19:11:00.522967 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:00.522956    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm" event={"ID":"82423fa5-f43a-4c81-ac33-df275d1bd095","Type":"ContainerDied","Data":"a98379bfa2363e3ab68a70095c5892fe1d8a49e084e2752c4710e047babe0b78"}
Oct 30 19:11:00.523282 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:00.523016    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a98379bfa2363e3ab68a70095c5892fe1d8a49e084e2752c4710e047babe0b78"
Oct 30 19:11:02.409304 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:02.409262    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:03.105722 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:03.105678    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:03.280933 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:03.280897    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:07.278641 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:07.278603    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz"]
Oct 30 19:11:07.670319 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:07.670289    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:07.850018 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:07.849982    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:09.048531 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:09.048490    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c736cb70-frgvv"]
Oct 30 19:11:09.094575 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:09.094541    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c736cb70-frgvv"]
Oct 30 19:11:10.845600 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:10.845558    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd776f26-6b23-44f8-be02-d58fa7960edf" path="/var/lib/kubelet/pods/cd776f26-6b23-44f8-be02-d58fa7960edf/volumes"
Oct 30 19:11:12.010296 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.010265    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:12.279168 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.279083    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz" podUID="363c9626-84ae-4605-95a3-7c000aebe1af" containerName="payload-processing" containerID="cri-o://e48476a0a92c7e93145c3c4d9f07ddede805cefcfedd80c9c7436978c4a2fd31" gracePeriod=25
Oct 30 19:11:12.512454 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.512424    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:12.526440 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.526401    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"]
Oct 30 19:11:12.527742 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.527705    4924 cpu_manager.go:414] "containerMap: removing container" podUID="82423fa5-f43a-4c81-ac33-df275d1bd095" containerName="revoke-keys"
Oct 30 19:11:12.527742 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.527730    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82423fa5-f43a-4c81-ac33-df275d1bd095" containerName="revoke-keys"
Oct 30 19:11:12.527948 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.527935    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="82423fa5-f43a-4c81-ac33-df275d1bd095" containerName="revoke-keys"
Oct 30 19:11:12.531903 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.531842    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.568593 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.568565    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz"
Oct 30 19:11:12.602151 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.602102    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"]
Oct 30 19:11:12.624932 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.624897    4924 generic.go:356] "Generic (PLEG): container finished" podID="363c9626-84ae-4605-95a3-7c000aebe1af" containerID="e48476a0a92c7e93145c3c4d9f07ddede805cefcfedd80c9c7436978c4a2fd31" exitCode=0
Oct 30 19:11:12.625221 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.625194    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz"
Oct 30 19:11:12.625408 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.625058    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz" event={"ID":"363c9626-84ae-4605-95a3-7c000aebe1af","Type":"ContainerDied","Data":"e48476a0a92c7e93145c3c4d9f07ddede805cefcfedd80c9c7436978c4a2fd31"}
Oct 30 19:11:12.625496 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.625459    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz" event={"ID":"363c9626-84ae-4605-95a3-7c000aebe1af","Type":"ContainerDied","Data":"b3c0dc2832561f9d58ec27427480e5ecb8f39e846bd131ff5cfcd067451c0ff4"}
Oct 30 19:11:12.625496 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.625489    4924 scope.go:122] "RemoveContainer" containerID="e48476a0a92c7e93145c3c4d9f07ddede805cefcfedd80c9c7436978c4a2fd31"
Oct 30 19:11:12.628469 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.628449    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-568107-openshift-default-dockercfg-9bbqg\""
Oct 30 19:11:12.632126 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.632081    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-credential-socket\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.634193 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.634166    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88w66\" (UniqueName: \"kubernetes.io/projected/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-kube-api-access-88w66\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.634326 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.634217    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-workload-socket\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.634401 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.634379    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-podinfo\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.634489 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.634470    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-envoy\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.634568 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.634559    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-token\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.634643 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.634593    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-data\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.634698 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.634640    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istiod-ca-cert\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.634698 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.634671    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-workload-certs\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.639945 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.639925    4924 scope.go:122] "RemoveContainer" containerID="e48476a0a92c7e93145c3c4d9f07ddede805cefcfedd80c9c7436978c4a2fd31"
Oct 30 19:11:12.640307 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:11:12.640282    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e48476a0a92c7e93145c3c4d9f07ddede805cefcfedd80c9c7436978c4a2fd31\": container with ID starting with e48476a0a92c7e93145c3c4d9f07ddede805cefcfedd80c9c7436978c4a2fd31 not found: ID does not exist" containerID="e48476a0a92c7e93145c3c4d9f07ddede805cefcfedd80c9c7436978c4a2fd31"
Oct 30 19:11:12.640419 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.640313    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e48476a0a92c7e93145c3c4d9f07ddede805cefcfedd80c9c7436978c4a2fd31"} err="failed to get container status \"e48476a0a92c7e93145c3c4d9f07ddede805cefcfedd80c9c7436978c4a2fd31\": rpc error: code = NotFound desc = could not find container \"e48476a0a92c7e93145c3c4d9f07ddede805cefcfedd80c9c7436978c4a2fd31\": container with ID starting with e48476a0a92c7e93145c3c4d9f07ddede805cefcfedd80c9c7436978c4a2fd31 not found: ID does not exist"
Oct 30 19:11:12.740128 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.739921    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:12.742683 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.741102    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363c9626-84ae-4605-95a3-7c000aebe1af-kube-api-access-8kvmh\" (UniqueName: \"kubernetes.io/projected/363c9626-84ae-4605-95a3-7c000aebe1af-kube-api-access-8kvmh\") pod \"363c9626-84ae-4605-95a3-7c000aebe1af\" (UID: \"363c9626-84ae-4605-95a3-7c000aebe1af\") "
Oct 30 19:11:12.742683 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.741154    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/363c9626-84ae-4605-95a3-7c000aebe1af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363c9626-84ae-4605-95a3-7c000aebe1af-plugins-config-volume\") pod \"363c9626-84ae-4605-95a3-7c000aebe1af\" (UID: \"363c9626-84ae-4605-95a3-7c000aebe1af\") "
Oct 30 19:11:12.742683 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.741489    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-workload-certs\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.742683 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.741548    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-credential-socket\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.742683 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.741579    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88w66\" (UniqueName: \"kubernetes.io/projected/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-kube-api-access-88w66\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.742683 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.741614    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-workload-socket\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.742683 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.741770    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-podinfo\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.742683 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.741877    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-envoy\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.742683 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.741953    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-token\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.742683 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.741986    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-data\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.742683 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.742033    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istiod-ca-cert\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.766092 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.759453    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-workload-socket\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.766092 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.765399    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-workload-certs\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.766092 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.765682    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-credential-socket\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.782147 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.774863    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-data\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.782147 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.777093    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363c9626-84ae-4605-95a3-7c000aebe1af-plugins-config-volume" pod "363c9626-84ae-4605-95a3-7c000aebe1af" (UID: "363c9626-84ae-4605-95a3-7c000aebe1af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:12.782147 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.777475    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istiod-ca-cert\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.791112 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.786671    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-envoy\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.791112 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.788637    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363c9626-84ae-4605-95a3-7c000aebe1af-kube-api-access-8kvmh" pod "363c9626-84ae-4605-95a3-7c000aebe1af" (UID: "363c9626-84ae-4605-95a3-7c000aebe1af"). InnerVolumeSpecName "kube-api-access-8kvmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:12.791112 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.789120    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-podinfo\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.825028 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.824951    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88w66\" (UniqueName: \"kubernetes.io/projected/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-kube-api-access-88w66\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.832026 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.831991    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-token\") pod \"e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") " pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:12.844348 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.844310    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kvmh\" (UniqueName: \"kubernetes.io/projected/363c9626-84ae-4605-95a3-7c000aebe1af-kube-api-access-8kvmh\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:12.844348 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.844346    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363c9626-84ae-4605-95a3-7c000aebe1af-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:12.867236 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:12.867189    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:13.038161 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:13.038081    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz"]
Oct 30 19:11:13.046018 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:13.045988    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cc079d-85df654c8d-fxctz"]
Oct 30 19:11:13.279566 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:13.279530    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:13.298512 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:13.298455    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"]
Oct 30 19:11:13.299226 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:11:13.299201    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01a04de_05cf_4bcc_83d3_a7d51ffc9977.slice/crio-0cd9d1a914a595113aad2dd490b99af37ebdb0981f08412757eb9280af0a0ca7 WatchSource:0}: Error finding container 0cd9d1a914a595113aad2dd490b99af37ebdb0981f08412757eb9280af0a0ca7: Status 404 returned error can't find the container with id 0cd9d1a914a595113aad2dd490b99af37ebdb0981f08412757eb9280af0a0ca7
Oct 30 19:11:13.302228 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:13.302196    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:11:13.302327 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:13.302294    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:11:13.302327 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:13.302324    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:11:13.635249 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:13.635212    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg" event={"ID":"c01a04de-05cf-4bcc-83d3-a7d51ffc9977","Type":"ContainerStarted","Data":"53aafde06247443f6080f5aa3cbe2add276430dd401b6d7a77c7d23da472b5e0"}
Oct 30 19:11:13.635438 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:13.635274    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg" event={"ID":"c01a04de-05cf-4bcc-83d3-a7d51ffc9977","Type":"ContainerStarted","Data":"0cd9d1a914a595113aad2dd490b99af37ebdb0981f08412757eb9280af0a0ca7"}
Oct 30 19:11:13.748560 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:13.748510    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg" podStartSLOduration=1.748493062 podStartE2EDuration="1.748493062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:13.744638946 +0000 UTC m=+2075.021689390" watchObservedRunningTime="2026-10-30 19:11:13.748493062 +0000 UTC m=+2075.025543506"
Oct 30 19:11:13.868778 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:13.868744    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:14.836717 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:14.836674    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="363c9626-84ae-4605-95a3-7c000aebe1af" path="/var/lib/kubelet/pods/363c9626-84ae-4605-95a3-7c000aebe1af/volumes"
Oct 30 19:11:14.873732 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:14.873703    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:15.652446 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:15.652418    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:15.653429 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:15.653410    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:11:15.713750 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:11:15.713716    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:11:16.876946 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:16.876901    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:17.066337 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:17.066305    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:17.409405 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:17.409371    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:17.783463 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:17.783193    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:20.590561 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:20.590514    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:20.886305 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:20.886264    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:21.099377 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:21.099333    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99"]
Oct 30 19:11:21.100103 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:21.100042    4924 cpu_manager.go:414] "containerMap: removing container" podUID="363c9626-84ae-4605-95a3-7c000aebe1af" containerName="payload-processing"
Oct 30 19:11:21.100103 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:21.100060    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="363c9626-84ae-4605-95a3-7c000aebe1af" containerName="payload-processing"
Oct 30 19:11:21.100279 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:21.100231    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="363c9626-84ae-4605-95a3-7c000aebe1af" containerName="payload-processing"
Oct 30 19:11:21.104912 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:21.104894    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99"
Oct 30 19:11:21.161996 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:21.161913    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99"]
Oct 30 19:11:21.238093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:21.199764    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d975d\" (UniqueName: \"kubernetes.io/projected/fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e-kube-api-access-d975d\") pod \"payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99\" (UID: \"fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99"
Oct 30 19:11:21.238093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:21.199907    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99\" (UID: \"fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99"
Oct 30 19:11:21.252088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:21.251542    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-568107\""
Oct 30 19:11:21.252088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:21.251678    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-568107-dockercfg-h6lr5\""
Oct 30 19:11:21.342259 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:21.342219    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d975d\" (UniqueName: \"kubernetes.io/projected/fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e-kube-api-access-d975d\") pod \"payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99\" (UID: \"fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99"
Oct 30 19:11:21.342636 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:21.342612    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99\" (UID: \"fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99"
Oct 30 19:11:21.343772 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:21.343748    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99\" (UID: \"fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99"
Oct 30 19:11:21.354701 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:21.354671    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l"]
Oct 30 19:11:21.359190 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:21.359171    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l"
Oct 30 19:11:21.398744 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:21.398716    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d975d\" (UniqueName: \"kubernetes.io/projected/fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e-kube-api-access-d975d\") pod \"payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99\" (UID: \"fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99"
Oct 30 19:11:21.409510 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:21.409477    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l"]
Oct 30 19:11:21.419539 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:21.419467    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99"
Oct 30 19:11:21.546969 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:21.546929    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65246d5a-1763-4c36-8744-0af6bec35f76-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l\" (UID: \"65246d5a-1763-4c36-8744-0af6bec35f76\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l"
Oct 30 19:11:21.547181 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:21.547089    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgmzh\" (UniqueName: \"kubernetes.io/projected/65246d5a-1763-4c36-8744-0af6bec35f76-kube-api-access-rgmzh\") pod \"payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l\" (UID: \"65246d5a-1763-4c36-8744-0af6bec35f76\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l"
Oct 30 19:11:21.648211 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:21.648176    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgmzh\" (UniqueName: \"kubernetes.io/projected/65246d5a-1763-4c36-8744-0af6bec35f76-kube-api-access-rgmzh\") pod \"payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l\" (UID: \"65246d5a-1763-4c36-8744-0af6bec35f76\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l"
Oct 30 19:11:21.648606 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:21.648488    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65246d5a-1763-4c36-8744-0af6bec35f76-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l\" (UID: \"65246d5a-1763-4c36-8744-0af6bec35f76\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l"
Oct 30 19:11:21.649229 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:21.649211    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65246d5a-1763-4c36-8744-0af6bec35f76-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l\" (UID: \"65246d5a-1763-4c36-8744-0af6bec35f76\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l"
Oct 30 19:11:21.708908 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:21.708809    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgmzh\" (UniqueName: \"kubernetes.io/projected/65246d5a-1763-4c36-8744-0af6bec35f76-kube-api-access-rgmzh\") pod \"payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l\" (UID: \"65246d5a-1763-4c36-8744-0af6bec35f76\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l"
Oct 30 19:11:21.795788 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:21.795741    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99"]
Oct 30 19:11:21.796299 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:11:21.796269    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb2b4a3a_d1c2_48f0_b451_f9b77b3d760e.slice/crio-506d5a8cf6e99a2337048db1d99fd5fb5439ae88d3a016c2cc85e3ef3b2db962 WatchSource:0}: Error finding container 506d5a8cf6e99a2337048db1d99fd5fb5439ae88d3a016c2cc85e3ef3b2db962: Status 404 returned error can't find the container with id 506d5a8cf6e99a2337048db1d99fd5fb5439ae88d3a016c2cc85e3ef3b2db962
Oct 30 19:11:21.972923 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:21.972844    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l"
Oct 30 19:11:22.352843 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:22.352814    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l"]
Oct 30 19:11:22.353111 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:11:22.353055    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65246d5a_1763_4c36_8744_0af6bec35f76.slice/crio-4d7ce13dc64d80dfa2f419e8fc613e901a466abffd9c294edf40f40a261ed569 WatchSource:0}: Error finding container 4d7ce13dc64d80dfa2f419e8fc613e901a466abffd9c294edf40f40a261ed569: Status 404 returned error can't find the container with id 4d7ce13dc64d80dfa2f419e8fc613e901a466abffd9c294edf40f40a261ed569
Oct 30 19:11:22.722001 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:22.721899    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l" event={"ID":"65246d5a-1763-4c36-8744-0af6bec35f76","Type":"ContainerStarted","Data":"643ba46f08edd2644883db388a8a892cae0009d4a86d4cf0ed008bc367cc939b"}
Oct 30 19:11:22.722001 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:22.721961    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l" event={"ID":"65246d5a-1763-4c36-8744-0af6bec35f76","Type":"ContainerStarted","Data":"4d7ce13dc64d80dfa2f419e8fc613e901a466abffd9c294edf40f40a261ed569"}
Oct 30 19:11:22.722001 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:22.721983    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l"
Oct 30 19:11:22.723727 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:22.723705    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99" event={"ID":"fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e","Type":"ContainerStarted","Data":"f5900b2635d9bcc16af5f060fb9904a9908bb26b86a03b79048f12c7237d6d98"}
Oct 30 19:11:22.723847 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:22.723743    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99" event={"ID":"fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e","Type":"ContainerStarted","Data":"506d5a8cf6e99a2337048db1d99fd5fb5439ae88d3a016c2cc85e3ef3b2db962"}
Oct 30 19:11:22.723847 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:22.723833    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99"
Oct 30 19:11:22.832704 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:22.832656    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l" podStartSLOduration=1.8326420140000002 podStartE2EDuration="1.832642014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:22.830308647 +0000 UTC m=+2084.107359086" watchObservedRunningTime="2026-10-30 19:11:22.832642014 +0000 UTC m=+2084.109692459"
Oct 30 19:11:22.944403 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:22.944335    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99" podStartSLOduration=1.9443177170000001 podStartE2EDuration="1.944317717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:22.940459104 +0000 UTC m=+2084.217509549" watchObservedRunningTime="2026-10-30 19:11:22.944317717 +0000 UTC m=+2084.221368162"
Oct 30 19:11:23.048323 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:23.048240    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:28.031460 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:28.031420    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:32.949080 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:32.949038    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:33.733495 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:33.733464    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99"
Oct 30 19:11:33.733649 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:33.733535    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l"
Oct 30 19:11:37.852156 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:37.852036    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:39.169313 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:39.169249    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 19:11:39.175184 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:39.175152    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 19:11:39.979412 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:39.979374    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:41.347996 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:41.347962    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z"]
Oct 30 19:11:41.353880 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:41.353855    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z"
Oct 30 19:11:41.364047 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:41.364019    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d6wd\" (UniqueName: \"kubernetes.io/projected/68e600a2-d100-4727-a613-cfc73ca09517-kube-api-access-2d6wd\") pod \"maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z\" (UID: \"68e600a2-d100-4727-a613-cfc73ca09517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z"
Oct 30 19:11:41.364289 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:41.364268    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68e600a2-d100-4727-a613-cfc73ca09517-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z\" (UID: \"68e600a2-d100-4727-a613-cfc73ca09517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z"
Oct 30 19:11:41.364399 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:41.364379    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68e600a2-d100-4727-a613-cfc73ca09517-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z\" (UID: \"68e600a2-d100-4727-a613-cfc73ca09517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z"
Oct 30 19:11:41.435196 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:41.435165    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z"]
Oct 30 19:11:41.444602 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:41.444571    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:11:41.449183 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:41.449155    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:11:41.449309 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:41.449280    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:11:41.465675 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:41.465651    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68e600a2-d100-4727-a613-cfc73ca09517-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z\" (UID: \"68e600a2-d100-4727-a613-cfc73ca09517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z"
Oct 30 19:11:41.465807 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:41.465707    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68e600a2-d100-4727-a613-cfc73ca09517-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z\" (UID: \"68e600a2-d100-4727-a613-cfc73ca09517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z"
Oct 30 19:11:41.465874 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:41.465809    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d6wd\" (UniqueName: \"kubernetes.io/projected/68e600a2-d100-4727-a613-cfc73ca09517-kube-api-access-2d6wd\") pod \"maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z\" (UID: \"68e600a2-d100-4727-a613-cfc73ca09517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z"
Oct 30 19:11:41.468204 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:41.468175    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68e600a2-d100-4727-a613-cfc73ca09517-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z\" (UID: \"68e600a2-d100-4727-a613-cfc73ca09517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z"
Oct 30 19:11:41.468927 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:41.468910    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:11:41.476648 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:41.476625    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68e600a2-d100-4727-a613-cfc73ca09517-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z\" (UID: \"68e600a2-d100-4727-a613-cfc73ca09517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z"
Oct 30 19:11:41.520473 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:41.520443    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d6wd\" (UniqueName: \"kubernetes.io/projected/68e600a2-d100-4727-a613-cfc73ca09517-kube-api-access-2d6wd\") pod \"maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z\" (UID: \"68e600a2-d100-4727-a613-cfc73ca09517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z"
Oct 30 19:11:41.668737 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:41.668697    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z"
Oct 30 19:11:42.105413 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:42.105375    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z"]
Oct 30 19:11:42.752612 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:42.752580    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:42.899175 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:42.899132    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z" event={"ID":"68e600a2-d100-4727-a613-cfc73ca09517","Type":"ContainerStarted","Data":"53e49aa02328855520565f071b92f2eab7b9229cbe331002eb2b75fc98dcf6c4"}
Oct 30 19:11:42.899373 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:42.899203    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z" event={"ID":"68e600a2-d100-4727-a613-cfc73ca09517","Type":"ContainerStarted","Data":"00a7b961ec33075254cdd03d78f72a602e29dffd5cdc98bdf4485b25f342974a"}
Oct 30 19:11:42.959672 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:42.959625    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:43.018110 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:43.010198    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z" podStartSLOduration=2.010178131 podStartE2EDuration="2.010178131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:43.008415884 +0000 UTC m=+2104.285466331" watchObservedRunningTime="2026-10-30 19:11:43.010178131 +0000 UTC m=+2104.287228577"
Oct 30 19:11:43.143437 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:43.143397    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:43.909080 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:43.909035    4924 generic.go:356] "Generic (PLEG): container finished" podID="68e600a2-d100-4727-a613-cfc73ca09517" containerID="53e49aa02328855520565f071b92f2eab7b9229cbe331002eb2b75fc98dcf6c4" exitCode=0
Oct 30 19:11:43.909557 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:43.909102    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z" event={"ID":"68e600a2-d100-4727-a613-cfc73ca09517","Type":"ContainerDied","Data":"53e49aa02328855520565f071b92f2eab7b9229cbe331002eb2b75fc98dcf6c4"}
Oct 30 19:11:43.948179 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:43.948151    4924 patch_prober.go:29] interesting pod/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:11:43.948341 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:43.948196    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk" podUID="34498bbd-c6eb-48b9-8003-d863dd955292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:11:45.095013 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:45.094985    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z"
Oct 30 19:11:45.130127 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:45.130052    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68e600a2-d100-4727-a613-cfc73ca09517-kube-api-access-2d6wd\" (UniqueName: \"kubernetes.io/projected/68e600a2-d100-4727-a613-cfc73ca09517-kube-api-access-2d6wd\") pod \"68e600a2-d100-4727-a613-cfc73ca09517\" (UID: \"68e600a2-d100-4727-a613-cfc73ca09517\") "
Oct 30 19:11:45.130288 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:45.130246    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68e600a2-d100-4727-a613-cfc73ca09517-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68e600a2-d100-4727-a613-cfc73ca09517-maas-api-openshift-ca\") pod \"68e600a2-d100-4727-a613-cfc73ca09517\" (UID: \"68e600a2-d100-4727-a613-cfc73ca09517\") "
Oct 30 19:11:45.130363 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:45.130289    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68e600a2-d100-4727-a613-cfc73ca09517-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68e600a2-d100-4727-a613-cfc73ca09517-maas-api-xks-ca\") pod \"68e600a2-d100-4727-a613-cfc73ca09517\" (UID: \"68e600a2-d100-4727-a613-cfc73ca09517\") "
Oct 30 19:11:45.130634 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:45.130607    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68e600a2-d100-4727-a613-cfc73ca09517-maas-api-openshift-ca" pod "68e600a2-d100-4727-a613-cfc73ca09517" (UID: "68e600a2-d100-4727-a613-cfc73ca09517"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:11:45.131043 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:45.131020    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68e600a2-d100-4727-a613-cfc73ca09517-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:45.132361 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:45.132336    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68e600a2-d100-4727-a613-cfc73ca09517-kube-api-access-2d6wd" pod "68e600a2-d100-4727-a613-cfc73ca09517" (UID: "68e600a2-d100-4727-a613-cfc73ca09517"). InnerVolumeSpecName "kube-api-access-2d6wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:45.132446 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:45.132342    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68e600a2-d100-4727-a613-cfc73ca09517-maas-api-xks-ca" pod "68e600a2-d100-4727-a613-cfc73ca09517" (UID: "68e600a2-d100-4727-a613-cfc73ca09517"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:11:45.234373 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:45.234285    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d6wd\" (UniqueName: \"kubernetes.io/projected/68e600a2-d100-4727-a613-cfc73ca09517-kube-api-access-2d6wd\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:45.234373 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:45.234326    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68e600a2-d100-4727-a613-cfc73ca09517-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:45.712613 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:11:45.712576    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:11:45.930199 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:45.930024    4924 generic.go:356] "Generic (PLEG): container finished" podID="34498bbd-c6eb-48b9-8003-d863dd955292" containerID="90b30198a6b8abc78e069f1c58737044cc9e9edc6e57e2517fd1e29f4a6a4921" exitCode=0
Oct 30 19:11:45.930445 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:45.930208    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk" event={"ID":"34498bbd-c6eb-48b9-8003-d863dd955292","Type":"ContainerDied","Data":"90b30198a6b8abc78e069f1c58737044cc9e9edc6e57e2517fd1e29f4a6a4921"}
Oct 30 19:11:45.933164 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:45.933139    4924 scope.go:122] "RemoveContainer" containerID="90b30198a6b8abc78e069f1c58737044cc9e9edc6e57e2517fd1e29f4a6a4921"
Oct 30 19:11:45.934474 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:45.934258    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z" event={"ID":"68e600a2-d100-4727-a613-cfc73ca09517","Type":"ContainerDied","Data":"00a7b961ec33075254cdd03d78f72a602e29dffd5cdc98bdf4485b25f342974a"}
Oct 30 19:11:45.934474 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:45.934312    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00a7b961ec33075254cdd03d78f72a602e29dffd5cdc98bdf4485b25f342974a"
Oct 30 19:11:45.934474 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:45.934370    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z"
Oct 30 19:11:45.935539 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:45.935508    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:11:45.935632 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:45.935608    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:11:45.935679 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:45.935642    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:11:46.955602 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:46.955568    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk" event={"ID":"34498bbd-c6eb-48b9-8003-d863dd955292","Type":"ContainerStarted","Data":"4b1ce5947ac907be160170c20aec19d95e72fab48fb600c92f2199961eaafd3f"}
Oct 30 19:11:47.327797 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:47.327697    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:47.496042 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:47.496007    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:11:47.496284 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:47.496247    4924 patch_prober.go:29] interesting pod/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 19:11:47.496355 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:47.496294    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk" podUID="34498bbd-c6eb-48b9-8003-d863dd955292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 19:11:48.496448 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:48.496413    4924 patch_prober.go:29] interesting pod/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 19:11:48.496855 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:48.496476    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk" podUID="34498bbd-c6eb-48b9-8003-d863dd955292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 19:11:49.142020 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:49.141974    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:49.329601 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:49.329566    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:49.496345 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:49.496261    4924 patch_prober.go:29] interesting pod/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 19:11:49.496345 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:49.496314    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk" podUID="34498bbd-c6eb-48b9-8003-d863dd955292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 19:11:49.571359 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:49.571325    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:49.647118 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:49.647078    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv"]
Oct 30 19:11:49.651814 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:49.651786    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-kv8sv"]
Oct 30 19:11:50.496401 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:50.496369    4924 patch_prober.go:29] interesting pod/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 19:11:50.496596 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:50.496423    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk" podUID="34498bbd-c6eb-48b9-8003-d863dd955292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 19:11:50.836652 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:50.836569    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8" path="/var/lib/kubelet/pods/9b3e536a-c40e-4fb9-8dde-c8e003c0b0d8/volumes"
Oct 30 19:11:51.373911 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:51.373879    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:51.495787 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:51.495748    4924 patch_prober.go:29] interesting pod/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 30 19:11:51.495996 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:51.495820    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk" podUID="34498bbd-c6eb-48b9-8003-d863dd955292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 30 19:11:51.567624 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:51.567558    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:52.500253 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:52.500221    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:11:52.500678 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:52.500655    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:11:52.501525 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:52.501506    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:11:53.813759 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:53.813720    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:54.005375 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:54.005343    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:54.194079 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:54.194043    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:54.396829 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:54.396756    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:55.038962 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:55.038898    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:55.231128 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:55.231095    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:58.094190 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:58.094154    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:58.304755 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:58.304725    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:11:59.262483 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:11:59.262445    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:00.347243 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:00.343929    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:00.530422 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:00.530388    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:00.635685 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:00.635580    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb"]
Oct 30 19:12:00.636741 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:00.636707    4924 cpu_manager.go:414] "containerMap: removing container" podUID="68e600a2-d100-4727-a613-cfc73ca09517" containerName="revoke-keys"
Oct 30 19:12:00.636741 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:00.636732    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68e600a2-d100-4727-a613-cfc73ca09517" containerName="revoke-keys"
Oct 30 19:12:00.636956 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:00.636943    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="68e600a2-d100-4727-a613-cfc73ca09517" containerName="revoke-keys"
Oct 30 19:12:00.640588 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:00.640569    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb"
Oct 30 19:12:00.692103 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:00.692076    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb"]
Oct 30 19:12:00.713699 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:00.713666    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbddd023-71f1-4a7b-90fb-ebfadb948013-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb\" (UID: \"fbddd023-71f1-4a7b-90fb-ebfadb948013\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb"
Oct 30 19:12:00.714137 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:00.714110    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdqzr\" (UniqueName: \"kubernetes.io/projected/fbddd023-71f1-4a7b-90fb-ebfadb948013-kube-api-access-bdqzr\") pod \"payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb\" (UID: \"fbddd023-71f1-4a7b-90fb-ebfadb948013\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb"
Oct 30 19:12:00.759273 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:00.759240    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-314bd6fa-dockercfg-xjbcq\""
Oct 30 19:12:00.812369 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:00.768420    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-314bd6fa\""
Oct 30 19:12:00.819772 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:00.819730    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbddd023-71f1-4a7b-90fb-ebfadb948013-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb\" (UID: \"fbddd023-71f1-4a7b-90fb-ebfadb948013\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb"
Oct 30 19:12:00.821395 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:00.821206    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbddd023-71f1-4a7b-90fb-ebfadb948013-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb\" (UID: \"fbddd023-71f1-4a7b-90fb-ebfadb948013\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb"
Oct 30 19:12:00.821709 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:00.821675    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdqzr\" (UniqueName: \"kubernetes.io/projected/fbddd023-71f1-4a7b-90fb-ebfadb948013-kube-api-access-bdqzr\") pod \"payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb\" (UID: \"fbddd023-71f1-4a7b-90fb-ebfadb948013\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb"
Oct 30 19:12:00.862024 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:00.861433    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl"]
Oct 30 19:12:00.873086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:00.870260    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl"]
Oct 30 19:12:00.873086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:00.870373    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl"
Oct 30 19:12:00.900824 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:00.900794    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdqzr\" (UniqueName: \"kubernetes.io/projected/fbddd023-71f1-4a7b-90fb-ebfadb948013-kube-api-access-bdqzr\") pod \"payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb\" (UID: \"fbddd023-71f1-4a7b-90fb-ebfadb948013\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb"
Oct 30 19:12:00.922986 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:00.922948    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91780c8b-9c6c-47bd-aa29-f85529dcd00f-plugins-config-volume\") pod \"payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl\" (UID: \"91780c8b-9c6c-47bd-aa29-f85529dcd00f\") " pod="openshift-ingress/payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl"
Oct 30 19:12:00.923256 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:00.923231    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkcrx\" (UniqueName: \"kubernetes.io/projected/91780c8b-9c6c-47bd-aa29-f85529dcd00f-kube-api-access-xkcrx\") pod \"payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl\" (UID: \"91780c8b-9c6c-47bd-aa29-f85529dcd00f\") " pod="openshift-ingress/payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl"
Oct 30 19:12:00.958516 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:00.958491    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb"
Oct 30 19:12:01.024891 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:01.024854    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91780c8b-9c6c-47bd-aa29-f85529dcd00f-plugins-config-volume\") pod \"payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl\" (UID: \"91780c8b-9c6c-47bd-aa29-f85529dcd00f\") " pod="openshift-ingress/payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl"
Oct 30 19:12:01.025124 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:01.025103    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkcrx\" (UniqueName: \"kubernetes.io/projected/91780c8b-9c6c-47bd-aa29-f85529dcd00f-kube-api-access-xkcrx\") pod \"payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl\" (UID: \"91780c8b-9c6c-47bd-aa29-f85529dcd00f\") " pod="openshift-ingress/payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl"
Oct 30 19:12:01.025928 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:01.025899    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91780c8b-9c6c-47bd-aa29-f85529dcd00f-plugins-config-volume\") pod \"payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl\" (UID: \"91780c8b-9c6c-47bd-aa29-f85529dcd00f\") " pod="openshift-ingress/payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl"
Oct 30 19:12:01.080226 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:01.080194    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkcrx\" (UniqueName: \"kubernetes.io/projected/91780c8b-9c6c-47bd-aa29-f85529dcd00f-kube-api-access-xkcrx\") pod \"payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl\" (UID: \"91780c8b-9c6c-47bd-aa29-f85529dcd00f\") " pod="openshift-ingress/payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl"
Oct 30 19:12:01.206933 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:01.206843    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl"
Oct 30 19:12:01.330526 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:01.330494    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb"]
Oct 30 19:12:01.332234 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:12:01.332198    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbddd023_71f1_4a7b_90fb_ebfadb948013.slice/crio-470fee6f5099718bb0c5158aa981f28f3dcc9bfb36d677b7866a2de6bf29e58b WatchSource:0}: Error finding container 470fee6f5099718bb0c5158aa981f28f3dcc9bfb36d677b7866a2de6bf29e58b: Status 404 returned error can't find the container with id 470fee6f5099718bb0c5158aa981f28f3dcc9bfb36d677b7866a2de6bf29e58b
Oct 30 19:12:01.576635 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:01.576609    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl"]
Oct 30 19:12:01.577227 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:12:01.577195    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91780c8b_9c6c_47bd_aa29_f85529dcd00f.slice/crio-fe9dbb022c6a861852b8bc523a64f567b7b72bef4693f6c1835a247a6c3ba9e1 WatchSource:0}: Error finding container fe9dbb022c6a861852b8bc523a64f567b7b72bef4693f6c1835a247a6c3ba9e1: Status 404 returned error can't find the container with id fe9dbb022c6a861852b8bc523a64f567b7b72bef4693f6c1835a247a6c3ba9e1
Oct 30 19:12:02.113050 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:02.113001    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl" event={"ID":"91780c8b-9c6c-47bd-aa29-f85529dcd00f","Type":"ContainerStarted","Data":"46dba677a96f0162bcc060bb1b6e6e9ea26fcf8209680d68f40a353999a887c9"}
Oct 30 19:12:02.119641 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:02.116141    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl"
Oct 30 19:12:02.119641 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:02.116193    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl" event={"ID":"91780c8b-9c6c-47bd-aa29-f85529dcd00f","Type":"ContainerStarted","Data":"fe9dbb022c6a861852b8bc523a64f567b7b72bef4693f6c1835a247a6c3ba9e1"}
Oct 30 19:12:02.119641 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:02.116220    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb" event={"ID":"fbddd023-71f1-4a7b-90fb-ebfadb948013","Type":"ContainerStarted","Data":"2c1d1397685929f5c647a6629c558c3917f096fc62164aa25964f4749fedc28b"}
Oct 30 19:12:02.119641 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:02.116243    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb" event={"ID":"fbddd023-71f1-4a7b-90fb-ebfadb948013","Type":"ContainerStarted","Data":"470fee6f5099718bb0c5158aa981f28f3dcc9bfb36d677b7866a2de6bf29e58b"}
Oct 30 19:12:02.119641 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:02.116269    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb"
Oct 30 19:12:02.224051 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:02.223997    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl" podStartSLOduration=2.223982439 podStartE2EDuration="2.223982439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:02.223261579 +0000 UTC m=+2123.500312026" watchObservedRunningTime="2026-10-30 19:12:02.223982439 +0000 UTC m=+2123.501032883"
Oct 30 19:12:02.353941 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:02.344413    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb" podStartSLOduration=2.344389836 podStartE2EDuration="2.344389836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:02.341374563 +0000 UTC m=+2123.618425009" watchObservedRunningTime="2026-10-30 19:12:02.344389836 +0000 UTC m=+2123.621440283"
Oct 30 19:12:04.862712 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:04.862673    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:10.304024 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:10.303986    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:13.139847 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:13.139818    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl"
Oct 30 19:12:13.140511 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:13.140492    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb"
Oct 30 19:12:13.216009 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:13.215983    4924 generic.go:356] "Generic (PLEG): container finished" podID="0c9fa098-37e6-48ad-ba03-1879f9017f7c" containerID="a2461b074d3d188c16e488a0dc09b2941276e4a70b34af07f3fd43c345633329" exitCode=0
Oct 30 19:12:13.216181 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:13.216050    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf" event={"ID":"0c9fa098-37e6-48ad-ba03-1879f9017f7c","Type":"ContainerDied","Data":"a2461b074d3d188c16e488a0dc09b2941276e4a70b34af07f3fd43c345633329"}
Oct 30 19:12:13.216418 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:13.216403    4924 scope.go:122] "RemoveContainer" containerID="a2461b074d3d188c16e488a0dc09b2941276e4a70b34af07f3fd43c345633329"
Oct 30 19:12:13.217977 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:13.217949    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:12:13.225935 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:13.218508    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:12:13.225935 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:13.218661    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:12:14.234762 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:14.234727    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf" event={"ID":"0c9fa098-37e6-48ad-ba03-1879f9017f7c","Type":"ContainerStarted","Data":"22b4c7af931de94782cbf1bde654e507b4be99e1a07af3ca9a39baa355e46197"}
Oct 30 19:12:14.701895 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:14.701864    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:12:14.702125 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:14.702037    4924 patch_prober.go:29] interesting pod/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 19:12:14.702125 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:14.702089    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf" podUID="0c9fa098-37e6-48ad-ba03-1879f9017f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 19:12:15.582787 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:15.582736    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:15.701760 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:15.701730    4924 patch_prober.go:29] interesting pod/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 19:12:15.701921 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:15.701791    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf" podUID="0c9fa098-37e6-48ad-ba03-1879f9017f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 19:12:15.737466 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:12:15.737425    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:12:16.705785 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:16.705359    4924 patch_prober.go:29] interesting pod/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 19:12:16.705785 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:16.705421    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf" podUID="0c9fa098-37e6-48ad-ba03-1879f9017f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 19:12:17.716088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:17.706363    4924 patch_prober.go:29] interesting pod/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 19:12:17.716088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:17.706423    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf" podUID="0c9fa098-37e6-48ad-ba03-1879f9017f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 19:12:18.713090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:18.702104    4924 patch_prober.go:29] interesting pod/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 30 19:12:18.713090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:18.702178    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf" podUID="0c9fa098-37e6-48ad-ba03-1879f9017f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 30 19:12:19.720361 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:19.720330    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:12:19.721090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:19.721054    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:12:19.721811 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:19.721795    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:12:20.154043 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:20.154003    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:21.867800 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:21.867767    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:22.157256 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:22.157035    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:22.316400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:22.316340    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cfb433-nr52q"]
Oct 30 19:12:22.357409 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:22.357381    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:22.362635 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:22.362603    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cfb433-nr52q"]
Oct 30 19:12:22.555682 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:22.555601    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:22.747853 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:22.747822    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:22.837388 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:22.837313    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="597ba804-ba56-4aa4-9f94-a95506856b98" path="/var/lib/kubelet/pods/597ba804-ba56-4aa4-9f94-a95506856b98/volumes"
Oct 30 19:12:22.930533 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:22.930499    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:26.604530 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:26.604498    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:28.276052 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:28.276018    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:28.462240 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:28.462208    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:28.646936 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:28.646890    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:28.748293 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:28.748257    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm"]
Oct 30 19:12:28.794587 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:28.794555    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cc079d-j69cm"]
Oct 30 19:12:28.836865 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:28.836833    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82423fa5-f43a-4c81-ac33-df275d1bd095" path="/var/lib/kubelet/pods/82423fa5-f43a-4c81-ac33-df275d1bd095/volumes"
Oct 30 19:12:29.216679 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:29.216649    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:29.417508 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:29.417473    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:29.604160 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:29.604128    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:29.818088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:29.810491    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:32.020564 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:32.020529    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:32.089367 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:32.089334    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2299c938-vn9l7"]
Oct 30 19:12:32.135859 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:32.135825    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2299c938-vn9l7"]
Oct 30 19:12:32.240081 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:32.240039    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:32.835847 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:32.835813    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3175550a-63b4-4493-a54c-1748e7626d88" path="/var/lib/kubelet/pods/3175550a-63b4-4493-a54c-1748e7626d88/volumes"
Oct 30 19:12:32.866848 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:32.866815    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"]
Oct 30 19:12:32.872416 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:32.872390    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:32.949602 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:32.949573    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f75d5f-openshift-default-dockercfg-dxbqq\""
Oct 30 19:12:32.958920 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:32.958888    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-credential-socket\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:32.959056 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:32.959004    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-workload-certs\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:32.959117 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:32.959054    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-data\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:32.959173 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:32.959160    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-envoy\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:32.959246 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:32.959234    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-workload-socket\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:32.959286 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:32.959258    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-token\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:32.959286 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:32.959282    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-podinfo\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:32.959369 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:32.959308    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrff4\" (UniqueName: \"kubernetes.io/projected/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-kube-api-access-wrff4\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:32.959458 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:32.959443    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istiod-ca-cert\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:32.984348 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:32.984314    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"]
Oct 30 19:12:33.060647 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:33.060613    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-credential-socket\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:33.061169 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:33.060707    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-workload-certs\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:33.061169 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:33.060766    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-data\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:33.061169 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:33.060846    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-envoy\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:33.061169 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:33.060907    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-workload-socket\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:33.061169 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:33.060936    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-token\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:33.061169 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:33.060974    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-podinfo\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:33.061169 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:33.061007    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrff4\" (UniqueName: \"kubernetes.io/projected/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-kube-api-access-wrff4\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:33.061169 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:33.061034    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-credential-socket\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:33.061710 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:33.061408    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-data\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:33.061710 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:33.061534    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-workload-socket\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:33.061935 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:33.061911    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istiod-ca-cert\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:33.062293 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:33.062268    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-workload-certs\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:33.062616 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:33.062592    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istiod-ca-cert\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:33.065661 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:33.065628    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-podinfo\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:33.067439 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:33.067415    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-envoy\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:33.117625 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:33.117567    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:33.120263 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:33.120244    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrff4\" (UniqueName: \"kubernetes.io/projected/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-kube-api-access-wrff4\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:33.120394 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:33.120373    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-token\") pod \"e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") " pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:33.187559 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:33.187533    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:33.505606 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:33.505505    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:33.606778 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:33.606747    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"]
Oct 30 19:12:33.641034 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:12:33.641008    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb1ef4a_ad74_421f_a5f7_f7e9de5bb421.slice/crio-e0df7bafac97a16096d052f8a6b0866f525c3b9cdea395b4955f3e004e77bbb7 WatchSource:0}: Error finding container e0df7bafac97a16096d052f8a6b0866f525c3b9cdea395b4955f3e004e77bbb7: Status 404 returned error can't find the container with id e0df7bafac97a16096d052f8a6b0866f525c3b9cdea395b4955f3e004e77bbb7
Oct 30 19:12:33.643993 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:33.643960    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:12:33.644094 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:33.644030    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:12:33.644094 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:33.644076    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:12:33.799992 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:33.799876    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:34.401459 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:34.401416    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745" event={"ID":"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421","Type":"ContainerStarted","Data":"0c9a2365f74bd1886974b2b5ae9c42ad508a8e8a50df847095f1382fcdfb7abe"}
Oct 30 19:12:34.401858 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:34.401483    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745" event={"ID":"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421","Type":"ContainerStarted","Data":"e0df7bafac97a16096d052f8a6b0866f525c3b9cdea395b4955f3e004e77bbb7"}
Oct 30 19:12:34.563547 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:34.563489    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745" podStartSLOduration=2.563472154 podStartE2EDuration="2.563472154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:34.56318158 +0000 UTC m=+2155.840232026" watchObservedRunningTime="2026-10-30 19:12:34.563472154 +0000 UTC m=+2155.840522600"
Oct 30 19:12:35.208824 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:35.208790    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:35.216139 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:35.216106    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:35.408553 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:35.408524    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:35.409498 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:35.409480    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:12:36.316186 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:36.316154    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:36.502493 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:36.502449    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:36.688795 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:36.688742    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:37.419513 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:37.419471    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:37.652800 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:37.652769    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:37.845016 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:37.844938    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:38.775765 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:38.775736    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd"]
Oct 30 19:12:38.780382 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:38.780362    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd"
Oct 30 19:12:38.853769 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:38.853733    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd"]
Oct 30 19:12:38.866244 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:38.866211    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca72b99-2e48-42dd-8a6b-cdcc54e153b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd\" (UID: \"9ca72b99-2e48-42dd-8a6b-cdcc54e153b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd"
Oct 30 19:12:38.867169 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:38.866324    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca72b99-2e48-42dd-8a6b-cdcc54e153b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd\" (UID: \"9ca72b99-2e48-42dd-8a6b-cdcc54e153b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd"
Oct 30 19:12:38.867169 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:38.866417    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wccjj\" (UniqueName: \"kubernetes.io/projected/9ca72b99-2e48-42dd-8a6b-cdcc54e153b6-kube-api-access-wccjj\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd\" (UID: \"9ca72b99-2e48-42dd-8a6b-cdcc54e153b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd"
Oct 30 19:12:38.881678 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:38.881643    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:12:38.881678 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:38.881665    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:12:38.881826 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:38.881675    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:12:38.881826 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:38.881705    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:12:38.967668 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:38.967632    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca72b99-2e48-42dd-8a6b-cdcc54e153b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd\" (UID: \"9ca72b99-2e48-42dd-8a6b-cdcc54e153b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd"
Oct 30 19:12:38.967821 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:38.967744    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wccjj\" (UniqueName: \"kubernetes.io/projected/9ca72b99-2e48-42dd-8a6b-cdcc54e153b6-kube-api-access-wccjj\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd\" (UID: \"9ca72b99-2e48-42dd-8a6b-cdcc54e153b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd"
Oct 30 19:12:38.967926 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:38.967908    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca72b99-2e48-42dd-8a6b-cdcc54e153b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd\" (UID: \"9ca72b99-2e48-42dd-8a6b-cdcc54e153b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd"
Oct 30 19:12:38.968540 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:38.968514    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca72b99-2e48-42dd-8a6b-cdcc54e153b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd\" (UID: \"9ca72b99-2e48-42dd-8a6b-cdcc54e153b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd"
Oct 30 19:12:38.970003 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:38.969982    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca72b99-2e48-42dd-8a6b-cdcc54e153b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd\" (UID: \"9ca72b99-2e48-42dd-8a6b-cdcc54e153b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd"
Oct 30 19:12:39.022922 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:39.022894    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wccjj\" (UniqueName: \"kubernetes.io/projected/9ca72b99-2e48-42dd-8a6b-cdcc54e153b6-kube-api-access-wccjj\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd\" (UID: \"9ca72b99-2e48-42dd-8a6b-cdcc54e153b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd"
Oct 30 19:12:39.095338 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:39.095268    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd"
Oct 30 19:12:39.442830 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:39.442722    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:39.487110 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:39.472774    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd"]
Oct 30 19:12:39.487110 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:12:39.482529    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca72b99_2e48_42dd_8a6b_cdcc54e153b6.slice/crio-5a8c6c6290090d0130f82765c3d4ef05b65e3e8963e606c1119c1988acb5e9e9 WatchSource:0}: Error finding container 5a8c6c6290090d0130f82765c3d4ef05b65e3e8963e606c1119c1988acb5e9e9: Status 404 returned error can't find the container with id 5a8c6c6290090d0130f82765c3d4ef05b65e3e8963e606c1119c1988acb5e9e9
Oct 30 19:12:40.304971 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:40.304938    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:40.448771 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:40.448744    4924 generic.go:356] "Generic (PLEG): container finished" podID="9ca72b99-2e48-42dd-8a6b-cdcc54e153b6" containerID="47632dd41219d5e9f816453f4807ddfd53f7c227856513a650fa3426cd368f1a" exitCode=0
Oct 30 19:12:40.448939 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:40.448819    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd" event={"ID":"9ca72b99-2e48-42dd-8a6b-cdcc54e153b6","Type":"ContainerDied","Data":"47632dd41219d5e9f816453f4807ddfd53f7c227856513a650fa3426cd368f1a"}
Oct 30 19:12:40.448939 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:40.448866    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd" event={"ID":"9ca72b99-2e48-42dd-8a6b-cdcc54e153b6","Type":"ContainerStarted","Data":"5a8c6c6290090d0130f82765c3d4ef05b65e3e8963e606c1119c1988acb5e9e9"}
Oct 30 19:12:40.490235 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:40.490201    4924 scope.go:122] "RemoveContainer" containerID="38ecb211ab7e6dae475bc9a3124ffbcb3f352de6d1fab2fae4fcc252e32522de"
Oct 30 19:12:40.500926 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:40.500903    4924 scope.go:122] "RemoveContainer" containerID="979b597cee473b38b283bdf82749226bc4955e3a6bbe5617f24f8cd553c62e97"
Oct 30 19:12:40.510579 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:40.510562    4924 scope.go:122] "RemoveContainer" containerID="6bc5455ea3e5342c66ecfce09c07f30d5c3e0d82fdf560f54272bafce8033a39"
Oct 30 19:12:40.519856 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:40.519835    4924 scope.go:122] "RemoveContainer" containerID="0f427cd2455c0cea4f3d0b5b0ded3c7f74a708e9f6bbba1a12f58d8ea8272b4e"
Oct 30 19:12:41.193767 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:41.193734    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:41.399885 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:41.399847    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:41.732140 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:41.725375    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd"
Oct 30 19:12:41.813189 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:41.798679    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd"]
Oct 30 19:12:41.813189 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:41.802044    4924 cpu_manager.go:414] "containerMap: removing container" podUID="9ca72b99-2e48-42dd-8a6b-cdcc54e153b6" containerName="revoke-keys"
Oct 30 19:12:41.813189 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:41.802093    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ca72b99-2e48-42dd-8a6b-cdcc54e153b6" containerName="revoke-keys"
Oct 30 19:12:41.813189 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:41.802415    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ca72b99-2e48-42dd-8a6b-cdcc54e153b6" containerName="revoke-keys"
Oct 30 19:12:41.813189 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:41.809924    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd"
Oct 30 19:12:41.817835 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:41.817416    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd"]
Oct 30 19:12:41.834607 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:41.834576    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ca72b99-2e48-42dd-8a6b-cdcc54e153b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca72b99-2e48-42dd-8a6b-cdcc54e153b6-maas-api-xks-ca\") pod \"9ca72b99-2e48-42dd-8a6b-cdcc54e153b6\" (UID: \"9ca72b99-2e48-42dd-8a6b-cdcc54e153b6\") "
Oct 30 19:12:41.834742 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:41.834696    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ca72b99-2e48-42dd-8a6b-cdcc54e153b6-kube-api-access-wccjj\" (UniqueName: \"kubernetes.io/projected/9ca72b99-2e48-42dd-8a6b-cdcc54e153b6-kube-api-access-wccjj\") pod \"9ca72b99-2e48-42dd-8a6b-cdcc54e153b6\" (UID: \"9ca72b99-2e48-42dd-8a6b-cdcc54e153b6\") "
Oct 30 19:12:41.834788 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:41.834760    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ca72b99-2e48-42dd-8a6b-cdcc54e153b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca72b99-2e48-42dd-8a6b-cdcc54e153b6-maas-api-openshift-ca\") pod \"9ca72b99-2e48-42dd-8a6b-cdcc54e153b6\" (UID: \"9ca72b99-2e48-42dd-8a6b-cdcc54e153b6\") "
Oct 30 19:12:41.835724 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:41.835696    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ca72b99-2e48-42dd-8a6b-cdcc54e153b6-maas-api-openshift-ca" pod "9ca72b99-2e48-42dd-8a6b-cdcc54e153b6" (UID: "9ca72b99-2e48-42dd-8a6b-cdcc54e153b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:41.839535 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:41.839506    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca72b99-2e48-42dd-8a6b-cdcc54e153b6-kube-api-access-wccjj" pod "9ca72b99-2e48-42dd-8a6b-cdcc54e153b6" (UID: "9ca72b99-2e48-42dd-8a6b-cdcc54e153b6"). InnerVolumeSpecName "kube-api-access-wccjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:41.841122 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:41.841059    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ca72b99-2e48-42dd-8a6b-cdcc54e153b6-maas-api-xks-ca" pod "9ca72b99-2e48-42dd-8a6b-cdcc54e153b6" (UID: "9ca72b99-2e48-42dd-8a6b-cdcc54e153b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:12:41.860544 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:41.859816    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f75d5f-dockercfg-6hn9v\""
Oct 30 19:12:41.860544 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:41.859818    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f75d5f\""
Oct 30 19:12:41.936186 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:41.936146    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn94j\" (UniqueName: \"kubernetes.io/projected/c05d45b0-ed8c-4682-872c-f8ac6e5b907e-kube-api-access-hn94j\") pod \"payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd\" (UID: \"c05d45b0-ed8c-4682-872c-f8ac6e5b907e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd"
Oct 30 19:12:41.936387 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:41.936356    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05d45b0-ed8c-4682-872c-f8ac6e5b907e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd\" (UID: \"c05d45b0-ed8c-4682-872c-f8ac6e5b907e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd"
Oct 30 19:12:41.936465 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:41.936452    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca72b99-2e48-42dd-8a6b-cdcc54e153b6-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:41.936511 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:41.936468    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wccjj\" (UniqueName: \"kubernetes.io/projected/9ca72b99-2e48-42dd-8a6b-cdcc54e153b6-kube-api-access-wccjj\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:41.936511 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:41.936478    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca72b99-2e48-42dd-8a6b-cdcc54e153b6-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:42.037790 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:42.037712    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05d45b0-ed8c-4682-872c-f8ac6e5b907e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd\" (UID: \"c05d45b0-ed8c-4682-872c-f8ac6e5b907e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd"
Oct 30 19:12:42.037933 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:42.037811    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn94j\" (UniqueName: \"kubernetes.io/projected/c05d45b0-ed8c-4682-872c-f8ac6e5b907e-kube-api-access-hn94j\") pod \"payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd\" (UID: \"c05d45b0-ed8c-4682-872c-f8ac6e5b907e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd"
Oct 30 19:12:42.038445 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:42.038424    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05d45b0-ed8c-4682-872c-f8ac6e5b907e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd\" (UID: \"c05d45b0-ed8c-4682-872c-f8ac6e5b907e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd"
Oct 30 19:12:42.092032 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:42.092002    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn94j\" (UniqueName: \"kubernetes.io/projected/c05d45b0-ed8c-4682-872c-f8ac6e5b907e-kube-api-access-hn94j\") pod \"payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd\" (UID: \"c05d45b0-ed8c-4682-872c-f8ac6e5b907e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd"
Oct 30 19:12:42.132217 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:42.132190    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd"
Oct 30 19:12:42.473008 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:42.472976    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd"
Oct 30 19:12:42.473410 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:42.473011    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd" event={"ID":"9ca72b99-2e48-42dd-8a6b-cdcc54e153b6","Type":"ContainerDied","Data":"5a8c6c6290090d0130f82765c3d4ef05b65e3e8963e606c1119c1988acb5e9e9"}
Oct 30 19:12:42.473410 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:42.473112    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a8c6c6290090d0130f82765c3d4ef05b65e3e8963e606c1119c1988acb5e9e9"
Oct 30 19:12:42.508493 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:42.508453    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd"]
Oct 30 19:12:42.508689 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:12:42.508659    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc05d45b0_ed8c_4682_872c_f8ac6e5b907e.slice/crio-cd4898c4bbe1727d3d04468eb642b2557112e80211a69fc6d23a9c06c516ce55 WatchSource:0}: Error finding container cd4898c4bbe1727d3d04468eb642b2557112e80211a69fc6d23a9c06c516ce55: Status 404 returned error can't find the container with id cd4898c4bbe1727d3d04468eb642b2557112e80211a69fc6d23a9c06c516ce55
Oct 30 19:12:43.483639 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:43.483607    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd" event={"ID":"c05d45b0-ed8c-4682-872c-f8ac6e5b907e","Type":"ContainerStarted","Data":"76be321eb639f1f7fff11c177aac30ba8f789ece27857e4a7597f21b2d92fedc"}
Oct 30 19:12:43.484033 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:43.483664    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd" event={"ID":"c05d45b0-ed8c-4682-872c-f8ac6e5b907e","Type":"ContainerStarted","Data":"cd4898c4bbe1727d3d04468eb642b2557112e80211a69fc6d23a9c06c516ce55"}
Oct 30 19:12:43.484033 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:43.483770    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd"
Oct 30 19:12:43.594702 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:43.594644    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd" podStartSLOduration=2.594631163 podStartE2EDuration="2.594631163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:43.591953309 +0000 UTC m=+2164.869003775" watchObservedRunningTime="2026-10-30 19:12:43.594631163 +0000 UTC m=+2164.871681608"
Oct 30 19:12:44.210220 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:44.210185    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05565d-j4bxr"]
Oct 30 19:12:44.215709 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:44.215685    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05565d-j4bxr"
Oct 30 19:12:44.269224 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:44.269164    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6000b1cb-428d-4beb-b0e5-3619a6a61096-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-05565d-j4bxr\" (UID: \"6000b1cb-428d-4beb-b0e5-3619a6a61096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05565d-j4bxr"
Oct 30 19:12:44.269585 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:44.269556    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6000b1cb-428d-4beb-b0e5-3619a6a61096-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-05565d-j4bxr\" (UID: \"6000b1cb-428d-4beb-b0e5-3619a6a61096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05565d-j4bxr"
Oct 30 19:12:44.269971 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:44.269631    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-642gc\" (UniqueName: \"kubernetes.io/projected/6000b1cb-428d-4beb-b0e5-3619a6a61096-kube-api-access-642gc\") pod \"maas-api-revoke-keys-e2e-api-a-05565d-j4bxr\" (UID: \"6000b1cb-428d-4beb-b0e5-3619a6a61096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05565d-j4bxr"
Oct 30 19:12:44.271538 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:44.271518    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05565d-j4bxr"]
Oct 30 19:12:44.311876 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:44.311848    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:12:44.312008 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:44.311966    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:12:44.312008 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:44.311988    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:12:44.312164 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:44.312011    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:12:44.371523 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:44.371492    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6000b1cb-428d-4beb-b0e5-3619a6a61096-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-05565d-j4bxr\" (UID: \"6000b1cb-428d-4beb-b0e5-3619a6a61096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05565d-j4bxr"
Oct 30 19:12:44.371708 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:44.371566    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6000b1cb-428d-4beb-b0e5-3619a6a61096-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-05565d-j4bxr\" (UID: \"6000b1cb-428d-4beb-b0e5-3619a6a61096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05565d-j4bxr"
Oct 30 19:12:44.371708 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:44.371632    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-642gc\" (UniqueName: \"kubernetes.io/projected/6000b1cb-428d-4beb-b0e5-3619a6a61096-kube-api-access-642gc\") pod \"maas-api-revoke-keys-e2e-api-a-05565d-j4bxr\" (UID: \"6000b1cb-428d-4beb-b0e5-3619a6a61096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05565d-j4bxr"
Oct 30 19:12:44.372328 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:44.372305    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6000b1cb-428d-4beb-b0e5-3619a6a61096-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-05565d-j4bxr\" (UID: \"6000b1cb-428d-4beb-b0e5-3619a6a61096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05565d-j4bxr"
Oct 30 19:12:44.374048 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:44.374019    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6000b1cb-428d-4beb-b0e5-3619a6a61096-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-05565d-j4bxr\" (UID: \"6000b1cb-428d-4beb-b0e5-3619a6a61096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05565d-j4bxr"
Oct 30 19:12:44.426683 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:44.426646    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-642gc\" (UniqueName: \"kubernetes.io/projected/6000b1cb-428d-4beb-b0e5-3619a6a61096-kube-api-access-642gc\") pod \"maas-api-revoke-keys-e2e-api-a-05565d-j4bxr\" (UID: \"6000b1cb-428d-4beb-b0e5-3619a6a61096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05565d-j4bxr"
Oct 30 19:12:44.534552 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:44.534465    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05565d-j4bxr"
Oct 30 19:12:44.794884 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:44.794796    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:44.929609 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:44.929469    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05565d-j4bxr"]
Oct 30 19:12:44.930360 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:12:44.930335    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6000b1cb_428d_4beb_b0e5_3619a6a61096.slice/crio-9c03f2152dee162b770da9c80daa427919a59b324cc410f405b4e03d387d8c61 WatchSource:0}: Error finding container 9c03f2152dee162b770da9c80daa427919a59b324cc410f405b4e03d387d8c61: Status 404 returned error can't find the container with id 9c03f2152dee162b770da9c80daa427919a59b324cc410f405b4e03d387d8c61
Oct 30 19:12:45.504829 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:45.504715    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05565d-j4bxr" event={"ID":"6000b1cb-428d-4beb-b0e5-3619a6a61096","Type":"ContainerStarted","Data":"30c6d97850af0a5860ed64fa0f0a597db1835dbbc06e3ef1570150eafc0985ba"}
Oct 30 19:12:45.504829 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:45.504783    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05565d-j4bxr" event={"ID":"6000b1cb-428d-4beb-b0e5-3619a6a61096","Type":"ContainerStarted","Data":"9c03f2152dee162b770da9c80daa427919a59b324cc410f405b4e03d387d8c61"}
Oct 30 19:12:45.615462 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:45.615388    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05565d-j4bxr" podStartSLOduration=1.6153727340000001 podStartE2EDuration="1.615372734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:45.611752754 +0000 UTC m=+2166.888803201" watchObservedRunningTime="2026-10-30 19:12:45.615372734 +0000 UTC m=+2166.892423180"
Oct 30 19:12:45.713180 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:12:45.713141    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:12:46.514311 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:46.514280    4924 generic.go:356] "Generic (PLEG): container finished" podID="6000b1cb-428d-4beb-b0e5-3619a6a61096" containerID="30c6d97850af0a5860ed64fa0f0a597db1835dbbc06e3ef1570150eafc0985ba" exitCode=0
Oct 30 19:12:46.514505 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:46.514349    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05565d-j4bxr" event={"ID":"6000b1cb-428d-4beb-b0e5-3619a6a61096","Type":"ContainerDied","Data":"30c6d97850af0a5860ed64fa0f0a597db1835dbbc06e3ef1570150eafc0985ba"}
Oct 30 19:12:47.699735 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:47.699710    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05565d-j4bxr"
Oct 30 19:12:47.831027 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:47.830948    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6000b1cb-428d-4beb-b0e5-3619a6a61096-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6000b1cb-428d-4beb-b0e5-3619a6a61096-maas-api-xks-ca\") pod \"6000b1cb-428d-4beb-b0e5-3619a6a61096\" (UID: \"6000b1cb-428d-4beb-b0e5-3619a6a61096\") "
Oct 30 19:12:47.831282 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:47.831261    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6000b1cb-428d-4beb-b0e5-3619a6a61096-kube-api-access-642gc\" (UniqueName: \"kubernetes.io/projected/6000b1cb-428d-4beb-b0e5-3619a6a61096-kube-api-access-642gc\") pod \"6000b1cb-428d-4beb-b0e5-3619a6a61096\" (UID: \"6000b1cb-428d-4beb-b0e5-3619a6a61096\") "
Oct 30 19:12:47.831345 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:47.831306    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6000b1cb-428d-4beb-b0e5-3619a6a61096-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6000b1cb-428d-4beb-b0e5-3619a6a61096-maas-api-openshift-ca\") pod \"6000b1cb-428d-4beb-b0e5-3619a6a61096\" (UID: \"6000b1cb-428d-4beb-b0e5-3619a6a61096\") "
Oct 30 19:12:47.831753 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:47.831725    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6000b1cb-428d-4beb-b0e5-3619a6a61096-maas-api-openshift-ca" pod "6000b1cb-428d-4beb-b0e5-3619a6a61096" (UID: "6000b1cb-428d-4beb-b0e5-3619a6a61096"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:47.832032 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:47.832011    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6000b1cb-428d-4beb-b0e5-3619a6a61096-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:47.833215 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:47.833190    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6000b1cb-428d-4beb-b0e5-3619a6a61096-maas-api-xks-ca" pod "6000b1cb-428d-4beb-b0e5-3619a6a61096" (UID: "6000b1cb-428d-4beb-b0e5-3619a6a61096"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:12:47.833883 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:47.833862    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6000b1cb-428d-4beb-b0e5-3619a6a61096-kube-api-access-642gc" pod "6000b1cb-428d-4beb-b0e5-3619a6a61096" (UID: "6000b1cb-428d-4beb-b0e5-3619a6a61096"). InnerVolumeSpecName "kube-api-access-642gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:47.933677 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:47.933640    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6000b1cb-428d-4beb-b0e5-3619a6a61096-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:47.933677 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:47.933674    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-642gc\" (UniqueName: \"kubernetes.io/projected/6000b1cb-428d-4beb-b0e5-3619a6a61096-kube-api-access-642gc\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:48.534702 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:48.534672    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05565d-j4bxr"
Oct 30 19:12:48.534702 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:48.534683    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05565d-j4bxr" event={"ID":"6000b1cb-428d-4beb-b0e5-3619a6a61096","Type":"ContainerDied","Data":"9c03f2152dee162b770da9c80daa427919a59b324cc410f405b4e03d387d8c61"}
Oct 30 19:12:48.534910 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:48.534733    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c03f2152dee162b770da9c80daa427919a59b324cc410f405b4e03d387d8c61"
Oct 30 19:12:50.086271 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:50.086240    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:50.271425 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:50.271311    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:50.463089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:50.462597    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:53.960831 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:53.960798    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr"]
Oct 30 19:12:54.064857 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:54.064823    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf"]
Oct 30 19:12:54.346614 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:54.346584    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:54.496881 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:54.496851    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd"
Oct 30 19:12:54.529267 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:54.529232    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:55.832591 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:55.832516    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:56.841577 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:56.841486    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:57.029597 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:57.029565    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:12:58.961720 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:58.961683    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr" podUID="b16e8a3e-7776-4313-a122-7ff9644f74d7" containerName="payload-processing" containerID="cri-o://1239c6b142b9875e046f26e96afcb591e2a3c233abaf535b61d92f6fcaf7e37a" gracePeriod=25
Oct 30 19:12:59.066091 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.066025    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf" podUID="6e1e3aa8-965d-4de6-86ef-e10a83d01fed" containerName="payload-pre-processing" containerID="cri-o://279f6e55cad0dceec45e7b300917d4eda403a72571b47e8b6aa98e568f6e53d5" gracePeriod=25
Oct 30 19:12:59.243139 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.243117    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr"
Oct 30 19:12:59.331905 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.331867    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b16e8a3e-7776-4313-a122-7ff9644f74d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16e8a3e-7776-4313-a122-7ff9644f74d7-plugins-config-volume\") pod \"b16e8a3e-7776-4313-a122-7ff9644f74d7\" (UID: \"b16e8a3e-7776-4313-a122-7ff9644f74d7\") "
Oct 30 19:12:59.332114 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.331940    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b16e8a3e-7776-4313-a122-7ff9644f74d7-kube-api-access-mrswz\" (UniqueName: \"kubernetes.io/projected/b16e8a3e-7776-4313-a122-7ff9644f74d7-kube-api-access-mrswz\") pod \"b16e8a3e-7776-4313-a122-7ff9644f74d7\" (UID: \"b16e8a3e-7776-4313-a122-7ff9644f74d7\") "
Oct 30 19:12:59.332315 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.332291    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b16e8a3e-7776-4313-a122-7ff9644f74d7-plugins-config-volume" pod "b16e8a3e-7776-4313-a122-7ff9644f74d7" (UID: "b16e8a3e-7776-4313-a122-7ff9644f74d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:59.332592 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.332575    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16e8a3e-7776-4313-a122-7ff9644f74d7-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:59.334172 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.334149    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b16e8a3e-7776-4313-a122-7ff9644f74d7-kube-api-access-mrswz" pod "b16e8a3e-7776-4313-a122-7ff9644f74d7" (UID: "b16e8a3e-7776-4313-a122-7ff9644f74d7"). InnerVolumeSpecName "kube-api-access-mrswz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:59.356156 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.356136    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf"
Oct 30 19:12:59.433629 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.433603    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e1e3aa8-965d-4de6-86ef-e10a83d01fed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e1e3aa8-965d-4de6-86ef-e10a83d01fed-plugins-config-volume\") pod \"6e1e3aa8-965d-4de6-86ef-e10a83d01fed\" (UID: \"6e1e3aa8-965d-4de6-86ef-e10a83d01fed\") "
Oct 30 19:12:59.433771 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.433712    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e1e3aa8-965d-4de6-86ef-e10a83d01fed-kube-api-access-h9nc9\" (UniqueName: \"kubernetes.io/projected/6e1e3aa8-965d-4de6-86ef-e10a83d01fed-kube-api-access-h9nc9\") pod \"6e1e3aa8-965d-4de6-86ef-e10a83d01fed\" (UID: \"6e1e3aa8-965d-4de6-86ef-e10a83d01fed\") "
Oct 30 19:12:59.434027 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.434004    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e1e3aa8-965d-4de6-86ef-e10a83d01fed-plugins-config-volume" pod "6e1e3aa8-965d-4de6-86ef-e10a83d01fed" (UID: "6e1e3aa8-965d-4de6-86ef-e10a83d01fed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:12:59.434357 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.434341    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrswz\" (UniqueName: \"kubernetes.io/projected/b16e8a3e-7776-4313-a122-7ff9644f74d7-kube-api-access-mrswz\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:59.434414 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.434360    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e1e3aa8-965d-4de6-86ef-e10a83d01fed-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:59.435751 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.435730    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1e3aa8-965d-4de6-86ef-e10a83d01fed-kube-api-access-h9nc9" pod "6e1e3aa8-965d-4de6-86ef-e10a83d01fed" (UID: "6e1e3aa8-965d-4de6-86ef-e10a83d01fed"). InnerVolumeSpecName "kube-api-access-h9nc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:59.536152 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.536056    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9nc9\" (UniqueName: \"kubernetes.io/projected/6e1e3aa8-965d-4de6-86ef-e10a83d01fed-kube-api-access-h9nc9\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:59.629259 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.629229    4924 generic.go:356] "Generic (PLEG): container finished" podID="6e1e3aa8-965d-4de6-86ef-e10a83d01fed" containerID="279f6e55cad0dceec45e7b300917d4eda403a72571b47e8b6aa98e568f6e53d5" exitCode=0
Oct 30 19:12:59.629438 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.629271    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf"
Oct 30 19:12:59.629438 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.629301    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf" event={"ID":"6e1e3aa8-965d-4de6-86ef-e10a83d01fed","Type":"ContainerDied","Data":"279f6e55cad0dceec45e7b300917d4eda403a72571b47e8b6aa98e568f6e53d5"}
Oct 30 19:12:59.629438 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.629352    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf" event={"ID":"6e1e3aa8-965d-4de6-86ef-e10a83d01fed","Type":"ContainerDied","Data":"4b579b20576c38132f02bc6d5631c5cb7021c6dbfe489be299769f638815b00f"}
Oct 30 19:12:59.629438 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.629374    4924 scope.go:122] "RemoveContainer" containerID="279f6e55cad0dceec45e7b300917d4eda403a72571b47e8b6aa98e568f6e53d5"
Oct 30 19:12:59.630974 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.630946    4924 generic.go:356] "Generic (PLEG): container finished" podID="b16e8a3e-7776-4313-a122-7ff9644f74d7" containerID="1239c6b142b9875e046f26e96afcb591e2a3c233abaf535b61d92f6fcaf7e37a" exitCode=0
Oct 30 19:12:59.631130 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.630973    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr" event={"ID":"b16e8a3e-7776-4313-a122-7ff9644f74d7","Type":"ContainerDied","Data":"1239c6b142b9875e046f26e96afcb591e2a3c233abaf535b61d92f6fcaf7e37a"}
Oct 30 19:12:59.631130 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.631022    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr" event={"ID":"b16e8a3e-7776-4313-a122-7ff9644f74d7","Type":"ContainerDied","Data":"cf627786042e3ebe77ede1c3bf3b69e3c2aa0c03b4740ed783258a16c64872e7"}
Oct 30 19:12:59.631130 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.630978    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr"
Oct 30 19:12:59.641827 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.641807    4924 scope.go:122] "RemoveContainer" containerID="279f6e55cad0dceec45e7b300917d4eda403a72571b47e8b6aa98e568f6e53d5"
Oct 30 19:12:59.642134 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:12:59.642103    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"279f6e55cad0dceec45e7b300917d4eda403a72571b47e8b6aa98e568f6e53d5\": container with ID starting with 279f6e55cad0dceec45e7b300917d4eda403a72571b47e8b6aa98e568f6e53d5 not found: ID does not exist" containerID="279f6e55cad0dceec45e7b300917d4eda403a72571b47e8b6aa98e568f6e53d5"
Oct 30 19:12:59.642254 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.642143    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"279f6e55cad0dceec45e7b300917d4eda403a72571b47e8b6aa98e568f6e53d5"} err="failed to get container status \"279f6e55cad0dceec45e7b300917d4eda403a72571b47e8b6aa98e568f6e53d5\": rpc error: code = NotFound desc = could not find container \"279f6e55cad0dceec45e7b300917d4eda403a72571b47e8b6aa98e568f6e53d5\": container with ID starting with 279f6e55cad0dceec45e7b300917d4eda403a72571b47e8b6aa98e568f6e53d5 not found: ID does not exist"
Oct 30 19:12:59.642254 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.642178    4924 scope.go:122] "RemoveContainer" containerID="1239c6b142b9875e046f26e96afcb591e2a3c233abaf535b61d92f6fcaf7e37a"
Oct 30 19:12:59.651530 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.651514    4924 scope.go:122] "RemoveContainer" containerID="1239c6b142b9875e046f26e96afcb591e2a3c233abaf535b61d92f6fcaf7e37a"
Oct 30 19:12:59.651765 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:12:59.651750    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1239c6b142b9875e046f26e96afcb591e2a3c233abaf535b61d92f6fcaf7e37a\": container with ID starting with 1239c6b142b9875e046f26e96afcb591e2a3c233abaf535b61d92f6fcaf7e37a not found: ID does not exist" containerID="1239c6b142b9875e046f26e96afcb591e2a3c233abaf535b61d92f6fcaf7e37a"
Oct 30 19:12:59.651812 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.651772    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1239c6b142b9875e046f26e96afcb591e2a3c233abaf535b61d92f6fcaf7e37a"} err="failed to get container status \"1239c6b142b9875e046f26e96afcb591e2a3c233abaf535b61d92f6fcaf7e37a\": rpc error: code = NotFound desc = could not find container \"1239c6b142b9875e046f26e96afcb591e2a3c233abaf535b61d92f6fcaf7e37a\": container with ID starting with 1239c6b142b9875e046f26e96afcb591e2a3c233abaf535b61d92f6fcaf7e37a not found: ID does not exist"
Oct 30 19:12:59.791846 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.791756    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr"]
Oct 30 19:12:59.796427 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.796400    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-3bd578-65d9f45f9b-h42lr"]
Oct 30 19:12:59.900058 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.900012    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf"]
Oct 30 19:12:59.906445 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:12:59.906421    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3bd578-84589c9c75-zjsdf"]
Oct 30 19:13:00.836700 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:00.836664    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e1e3aa8-965d-4de6-86ef-e10a83d01fed" path="/var/lib/kubelet/pods/6e1e3aa8-965d-4de6-86ef-e10a83d01fed/volumes"
Oct 30 19:13:00.837277 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:00.837257    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b16e8a3e-7776-4313-a122-7ff9644f74d7" path="/var/lib/kubelet/pods/b16e8a3e-7776-4313-a122-7ff9644f74d7/volumes"
Oct 30 19:13:01.020156 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:01.020122    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:05.865540 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:05.865499    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:08.371127 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:08.371086    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9"]
Oct 30 19:13:08.371817 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:08.371797    4924 cpu_manager.go:414] "containerMap: removing container" podUID="6000b1cb-428d-4beb-b0e5-3619a6a61096" containerName="revoke-keys"
Oct 30 19:13:08.371914 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:08.371819    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6000b1cb-428d-4beb-b0e5-3619a6a61096" containerName="revoke-keys"
Oct 30 19:13:08.371914 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:08.371879    4924 cpu_manager.go:414] "containerMap: removing container" podUID="6e1e3aa8-965d-4de6-86ef-e10a83d01fed" containerName="payload-pre-processing"
Oct 30 19:13:08.371914 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:08.371888    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e1e3aa8-965d-4de6-86ef-e10a83d01fed" containerName="payload-pre-processing"
Oct 30 19:13:08.372107 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:08.371944    4924 cpu_manager.go:414] "containerMap: removing container" podUID="b16e8a3e-7776-4313-a122-7ff9644f74d7" containerName="payload-processing"
Oct 30 19:13:08.372107 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:08.371952    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b16e8a3e-7776-4313-a122-7ff9644f74d7" containerName="payload-processing"
Oct 30 19:13:08.372210 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:08.372139    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="b16e8a3e-7776-4313-a122-7ff9644f74d7" containerName="payload-processing"
Oct 30 19:13:08.372210 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:08.372154    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="6000b1cb-428d-4beb-b0e5-3619a6a61096" containerName="revoke-keys"
Oct 30 19:13:08.372210 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:08.372169    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e1e3aa8-965d-4de6-86ef-e10a83d01fed" containerName="payload-pre-processing"
Oct 30 19:13:08.375711 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:08.375695    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9"
Oct 30 19:13:08.426340 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:08.426297    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:13:08.426481 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:08.426374    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:13:08.430639 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:08.430616    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9"]
Oct 30 19:13:08.463625 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:08.463592    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ced340-da28-45e1-867f-de3d1ff4017a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9\" (UID: \"36ced340-da28-45e1-867f-de3d1ff4017a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9"
Oct 30 19:13:08.463802 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:08.463782    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbbhg\" (UniqueName: \"kubernetes.io/projected/36ced340-da28-45e1-867f-de3d1ff4017a-kube-api-access-gbbhg\") pod \"maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9\" (UID: \"36ced340-da28-45e1-867f-de3d1ff4017a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9"
Oct 30 19:13:08.463874 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:08.463826    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ced340-da28-45e1-867f-de3d1ff4017a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9\" (UID: \"36ced340-da28-45e1-867f-de3d1ff4017a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9"
Oct 30 19:13:08.472489 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:08.472464    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:13:08.472575 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:08.472485    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:13:08.565047 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:08.565019    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbbhg\" (UniqueName: \"kubernetes.io/projected/36ced340-da28-45e1-867f-de3d1ff4017a-kube-api-access-gbbhg\") pod \"maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9\" (UID: \"36ced340-da28-45e1-867f-de3d1ff4017a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9"
Oct 30 19:13:08.565230 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:08.565060    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ced340-da28-45e1-867f-de3d1ff4017a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9\" (UID: \"36ced340-da28-45e1-867f-de3d1ff4017a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9"
Oct 30 19:13:08.565305 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:08.565245    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ced340-da28-45e1-867f-de3d1ff4017a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9\" (UID: \"36ced340-da28-45e1-867f-de3d1ff4017a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9"
Oct 30 19:13:08.565829 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:08.565811    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ced340-da28-45e1-867f-de3d1ff4017a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9\" (UID: \"36ced340-da28-45e1-867f-de3d1ff4017a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9"
Oct 30 19:13:08.567569 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:08.567550    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ced340-da28-45e1-867f-de3d1ff4017a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9\" (UID: \"36ced340-da28-45e1-867f-de3d1ff4017a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9"
Oct 30 19:13:08.620197 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:08.620166    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbbhg\" (UniqueName: \"kubernetes.io/projected/36ced340-da28-45e1-867f-de3d1ff4017a-kube-api-access-gbbhg\") pod \"maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9\" (UID: \"36ced340-da28-45e1-867f-de3d1ff4017a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9"
Oct 30 19:13:08.689635 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:08.689563    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9"
Oct 30 19:13:09.072057 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:09.071944    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9"]
Oct 30 19:13:09.073249 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:13:09.073218    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ced340_da28_45e1_867f_de3d1ff4017a.slice/crio-89ff324974184c4f68c6e20f0151704a3fd63ba781ae0dc607f0b7ff9af026cf WatchSource:0}: Error finding container 89ff324974184c4f68c6e20f0151704a3fd63ba781ae0dc607f0b7ff9af026cf: Status 404 returned error can't find the container with id 89ff324974184c4f68c6e20f0151704a3fd63ba781ae0dc607f0b7ff9af026cf
Oct 30 19:13:09.708199 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:09.708160    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9" event={"ID":"36ced340-da28-45e1-867f-de3d1ff4017a","Type":"ContainerStarted","Data":"101cde382f9a3c71f8b44f7c7635c4f7bde7e900cf26c9f298d1810d898b6d59"}
Oct 30 19:13:09.708564 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:09.708223    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9" event={"ID":"36ced340-da28-45e1-867f-de3d1ff4017a","Type":"ContainerStarted","Data":"89ff324974184c4f68c6e20f0151704a3fd63ba781ae0dc607f0b7ff9af026cf"}
Oct 30 19:13:10.716454 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:10.716422    4924 generic.go:356] "Generic (PLEG): container finished" podID="36ced340-da28-45e1-867f-de3d1ff4017a" containerID="101cde382f9a3c71f8b44f7c7635c4f7bde7e900cf26c9f298d1810d898b6d59" exitCode=0
Oct 30 19:13:10.716835 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:10.716475    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9" event={"ID":"36ced340-da28-45e1-867f-de3d1ff4017a","Type":"ContainerDied","Data":"101cde382f9a3c71f8b44f7c7635c4f7bde7e900cf26c9f298d1810d898b6d59"}
Oct 30 19:13:10.778953 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:10.778895    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:11.900984 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:11.900957    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9"
Oct 30 19:13:12.013379 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:12.013294    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ced340-da28-45e1-867f-de3d1ff4017a-kube-api-access-gbbhg\" (UniqueName: \"kubernetes.io/projected/36ced340-da28-45e1-867f-de3d1ff4017a-kube-api-access-gbbhg\") pod \"36ced340-da28-45e1-867f-de3d1ff4017a\" (UID: \"36ced340-da28-45e1-867f-de3d1ff4017a\") "
Oct 30 19:13:12.013562 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:12.013419    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36ced340-da28-45e1-867f-de3d1ff4017a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ced340-da28-45e1-867f-de3d1ff4017a-maas-api-xks-ca\") pod \"36ced340-da28-45e1-867f-de3d1ff4017a\" (UID: \"36ced340-da28-45e1-867f-de3d1ff4017a\") "
Oct 30 19:13:12.013562 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:12.013470    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36ced340-da28-45e1-867f-de3d1ff4017a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ced340-da28-45e1-867f-de3d1ff4017a-maas-api-openshift-ca\") pod \"36ced340-da28-45e1-867f-de3d1ff4017a\" (UID: \"36ced340-da28-45e1-867f-de3d1ff4017a\") "
Oct 30 19:13:12.014051 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:12.014018    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ced340-da28-45e1-867f-de3d1ff4017a-maas-api-openshift-ca" pod "36ced340-da28-45e1-867f-de3d1ff4017a" (UID: "36ced340-da28-45e1-867f-de3d1ff4017a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:12.014367 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:12.014284    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ced340-da28-45e1-867f-de3d1ff4017a-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:12.015804 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:12.015782    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ced340-da28-45e1-867f-de3d1ff4017a-kube-api-access-gbbhg" pod "36ced340-da28-45e1-867f-de3d1ff4017a" (UID: "36ced340-da28-45e1-867f-de3d1ff4017a"). InnerVolumeSpecName "kube-api-access-gbbhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:12.015899 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:12.015880    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ced340-da28-45e1-867f-de3d1ff4017a-maas-api-xks-ca" pod "36ced340-da28-45e1-867f-de3d1ff4017a" (UID: "36ced340-da28-45e1-867f-de3d1ff4017a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:13:12.115973 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:12.115944    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbbhg\" (UniqueName: \"kubernetes.io/projected/36ced340-da28-45e1-867f-de3d1ff4017a-kube-api-access-gbbhg\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:12.115973 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:12.115971    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ced340-da28-45e1-867f-de3d1ff4017a-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:12.732882 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:12.732850    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9"
Oct 30 19:13:12.733050 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:12.732883    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9" event={"ID":"36ced340-da28-45e1-867f-de3d1ff4017a","Type":"ContainerDied","Data":"89ff324974184c4f68c6e20f0151704a3fd63ba781ae0dc607f0b7ff9af026cf"}
Oct 30 19:13:12.733050 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:12.732958    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89ff324974184c4f68c6e20f0151704a3fd63ba781ae0dc607f0b7ff9af026cf"
Oct 30 19:13:15.506106 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:15.506055    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"]
Oct 30 19:13:15.506584 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:15.506285    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5" podUID="0e31d430-f842-4ba1-9890-6b1000521b5d" containerName="istio-proxy" containerID="cri-o://d98742fb6ef83fe326b8650726d2dce31df92d65e17f37d4920c5e85d7b4f802" gracePeriod=30
Oct 30 19:13:15.535766 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:15.535682    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:15.715166 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:13:15.715131    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:13:15.742722 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:15.742663    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"]
Oct 30 19:13:15.744045 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:15.742938    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl" podUID="4947eb13-56e8-4d54-b253-21913852aeb1" containerName="istio-proxy" containerID="cri-o://16be12feb68fac8f7a9066d85e368b7d706ffe6d221c3583b9fc937cee169362" gracePeriod=30
Oct 30 19:13:15.747569 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:15.747542    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"]
Oct 30 19:13:15.747813 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:15.747783    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh" podUID="971fec4f-5a07-4eae-b318-0e9c8f381983" containerName="istio-proxy" containerID="cri-o://57e1d57c7966a5575a5dfe2d548d6e30d4b99a905147c2f995c80e275c47bb63" gracePeriod=30
Oct 30 19:13:17.186502 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:17.186470    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:17.474414 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:17.474326    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:17.658497 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:17.658461    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:17.665355 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:17.665321    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z"]
Oct 30 19:13:17.710352 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:17.710323    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-071cf0-5kr8z"]
Oct 30 19:13:18.017766 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:18.017731    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:18.196323 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:18.196290    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:18.595447 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:18.595406    4924 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:13:18.595642 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:18.595471    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh" podUID="971fec4f-5a07-4eae-b318-0e9c8f381983" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:13:18.843772 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:18.843734    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68e600a2-d100-4727-a613-cfc73ca09517" path="/var/lib/kubelet/pods/68e600a2-d100-4727-a613-cfc73ca09517/volumes"
Oct 30 19:13:18.844750 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:18.844724    4924 patch_prober.go:29] interesting pod/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:13:18.844887 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:18.844770    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl" podUID="4947eb13-56e8-4d54-b253-21913852aeb1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:13:18.965484 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:18.965455    4924 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:13:18.965647 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:18.965500    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5" podUID="0e31d430-f842-4ba1-9890-6b1000521b5d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:13:20.125026 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.124931    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:20.311360 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.311327    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:20.798680 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.798652    4924 generic.go:356] "Generic (PLEG): container finished" podID="0e31d430-f842-4ba1-9890-6b1000521b5d" containerID="d98742fb6ef83fe326b8650726d2dce31df92d65e17f37d4920c5e85d7b4f802" exitCode=0
Oct 30 19:13:20.798783 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.798720    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5" event={"ID":"0e31d430-f842-4ba1-9890-6b1000521b5d","Type":"ContainerDied","Data":"d98742fb6ef83fe326b8650726d2dce31df92d65e17f37d4920c5e85d7b4f802"}
Oct 30 19:13:20.915424 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.915399    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 19:13:20.969377 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.969350    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-token\" (UniqueName: \"kubernetes.io/projected/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-token\") pod \"0e31d430-f842-4ba1-9890-6b1000521b5d\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") "
Oct 30 19:13:20.969629 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.969486    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e31d430-f842-4ba1-9890-6b1000521b5d-kube-api-access-qkwdn\" (UniqueName: \"kubernetes.io/projected/0e31d430-f842-4ba1-9890-6b1000521b5d-kube-api-access-qkwdn\") pod \"0e31d430-f842-4ba1-9890-6b1000521b5d\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") "
Oct 30 19:13:20.969629 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.969524    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-envoy\") pod \"0e31d430-f842-4ba1-9890-6b1000521b5d\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") "
Oct 30 19:13:20.970390 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.970364    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-workload-certs\") pod \"0e31d430-f842-4ba1-9890-6b1000521b5d\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") "
Oct 30 19:13:20.970638 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.970621    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-workload-socket\") pod \"0e31d430-f842-4ba1-9890-6b1000521b5d\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") "
Oct 30 19:13:20.970738 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.970632    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-workload-certs" pod "0e31d430-f842-4ba1-9890-6b1000521b5d" (UID: "0e31d430-f842-4ba1-9890-6b1000521b5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:20.970896 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.970877    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-workload-socket" pod "0e31d430-f842-4ba1-9890-6b1000521b5d" (UID: "0e31d430-f842-4ba1-9890-6b1000521b5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:20.971131 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.971111    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-data\") pod \"0e31d430-f842-4ba1-9890-6b1000521b5d\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") "
Oct 30 19:13:20.971385 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.971368    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-data" pod "0e31d430-f842-4ba1-9890-6b1000521b5d" (UID: "0e31d430-f842-4ba1-9890-6b1000521b5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:20.971472 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.971419    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-podinfo\") pod \"0e31d430-f842-4ba1-9890-6b1000521b5d\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") "
Oct 30 19:13:20.971726 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.971705    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-credential-socket" pod "0e31d430-f842-4ba1-9890-6b1000521b5d" (UID: "0e31d430-f842-4ba1-9890-6b1000521b5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:20.971819 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.971798    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-token" pod "0e31d430-f842-4ba1-9890-6b1000521b5d" (UID: "0e31d430-f842-4ba1-9890-6b1000521b5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:20.971887 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.971454    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-credential-socket\") pod \"0e31d430-f842-4ba1-9890-6b1000521b5d\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") "
Oct 30 19:13:20.971953 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.971928    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e31d430-f842-4ba1-9890-6b1000521b5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e31d430-f842-4ba1-9890-6b1000521b5d-istiod-ca-cert\") pod \"0e31d430-f842-4ba1-9890-6b1000521b5d\" (UID: \"0e31d430-f842-4ba1-9890-6b1000521b5d\") "
Oct 30 19:13:20.972274 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.972255    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e31d430-f842-4ba1-9890-6b1000521b5d-istiod-ca-cert" pod "0e31d430-f842-4ba1-9890-6b1000521b5d" (UID: "0e31d430-f842-4ba1-9890-6b1000521b5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:20.972798 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.972771    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e31d430-f842-4ba1-9890-6b1000521b5d-kube-api-access-qkwdn" pod "0e31d430-f842-4ba1-9890-6b1000521b5d" (UID: "0e31d430-f842-4ba1-9890-6b1000521b5d"). InnerVolumeSpecName "kube-api-access-qkwdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:20.972798 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.972795    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:20.972973 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.972814    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:20.972973 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.972825    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:20.972973 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.972834    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:20.972973 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.972842    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:20.972973 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.972850    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e31d430-f842-4ba1-9890-6b1000521b5d-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:20.973697 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.973676    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-podinfo" pod "0e31d430-f842-4ba1-9890-6b1000521b5d" (UID: "0e31d430-f842-4ba1-9890-6b1000521b5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:13:20.974550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:20.974523    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-envoy" pod "0e31d430-f842-4ba1-9890-6b1000521b5d" (UID: "0e31d430-f842-4ba1-9890-6b1000521b5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.036572 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.036508    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 19:13:21.151105 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.149525    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.151105 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.149555    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkwdn\" (UniqueName: \"kubernetes.io/projected/0e31d430-f842-4ba1-9890-6b1000521b5d-kube-api-access-qkwdn\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.151105 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.149573    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e31d430-f842-4ba1-9890-6b1000521b5d-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.157138 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.156319    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 19:13:21.250549 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.250518    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-workload-certs\") pod \"971fec4f-5a07-4eae-b318-0e9c8f381983\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") "
Oct 30 19:13:21.250720 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.250584    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-istio-envoy\") pod \"4947eb13-56e8-4d54-b253-21913852aeb1\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") "
Oct 30 19:13:21.250720 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.250632    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-workload-socket\") pod \"971fec4f-5a07-4eae-b318-0e9c8f381983\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") "
Oct 30 19:13:21.250720 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.250677    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/971fec4f-5a07-4eae-b318-0e9c8f381983-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/971fec4f-5a07-4eae-b318-0e9c8f381983-istiod-ca-cert\") pod \"971fec4f-5a07-4eae-b318-0e9c8f381983\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") "
Oct 30 19:13:21.250720 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.250710    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-workload-certs\") pod \"4947eb13-56e8-4d54-b253-21913852aeb1\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") "
Oct 30 19:13:21.250952 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.250752    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-credential-socket\") pod \"4947eb13-56e8-4d54-b253-21913852aeb1\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") "
Oct 30 19:13:21.250952 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.250788    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4947eb13-56e8-4d54-b253-21913852aeb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4947eb13-56e8-4d54-b253-21913852aeb1-istiod-ca-cert\") pod \"4947eb13-56e8-4d54-b253-21913852aeb1\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") "
Oct 30 19:13:21.250952 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.250830    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-workload-certs" pod "971fec4f-5a07-4eae-b318-0e9c8f381983" (UID: "971fec4f-5a07-4eae-b318-0e9c8f381983"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.250952 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.250844    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4947eb13-56e8-4d54-b253-21913852aeb1-istio-token\" (UniqueName: \"kubernetes.io/projected/4947eb13-56e8-4d54-b253-21913852aeb1-istio-token\") pod \"4947eb13-56e8-4d54-b253-21913852aeb1\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") "
Oct 30 19:13:21.250952 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.250936    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-workload-socket\") pod \"4947eb13-56e8-4d54-b253-21913852aeb1\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") "
Oct 30 19:13:21.251257 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.250961    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-workload-socket" pod "971fec4f-5a07-4eae-b318-0e9c8f381983" (UID: "971fec4f-5a07-4eae-b318-0e9c8f381983"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.251257 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.251006    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-istio-data\") pod \"4947eb13-56e8-4d54-b253-21913852aeb1\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") "
Oct 30 19:13:21.251257 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.251057    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-credential-socket\") pod \"971fec4f-5a07-4eae-b318-0e9c8f381983\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") "
Oct 30 19:13:21.251257 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.251117    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-credential-socket" pod "4947eb13-56e8-4d54-b253-21913852aeb1" (UID: "4947eb13-56e8-4d54-b253-21913852aeb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.251257 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.251161    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-envoy\") pod \"971fec4f-5a07-4eae-b318-0e9c8f381983\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") "
Oct 30 19:13:21.251257 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.251239    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-workload-socket" pod "4947eb13-56e8-4d54-b253-21913852aeb1" (UID: "4947eb13-56e8-4d54-b253-21913852aeb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.251257 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.251245    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4947eb13-56e8-4d54-b253-21913852aeb1-kube-api-access-bvgn5\" (UniqueName: \"kubernetes.io/projected/4947eb13-56e8-4d54-b253-21913852aeb1-kube-api-access-bvgn5\") pod \"4947eb13-56e8-4d54-b253-21913852aeb1\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") "
Oct 30 19:13:21.251616 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.251307    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4947eb13-56e8-4d54-b253-21913852aeb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4947eb13-56e8-4d54-b253-21913852aeb1-istio-podinfo\") pod \"4947eb13-56e8-4d54-b253-21913852aeb1\" (UID: \"4947eb13-56e8-4d54-b253-21913852aeb1\") "
Oct 30 19:13:21.251616 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.251357    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-istio-data" pod "4947eb13-56e8-4d54-b253-21913852aeb1" (UID: "4947eb13-56e8-4d54-b253-21913852aeb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.251616 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.251354    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4947eb13-56e8-4d54-b253-21913852aeb1-istiod-ca-cert" pod "4947eb13-56e8-4d54-b253-21913852aeb1" (UID: "4947eb13-56e8-4d54-b253-21913852aeb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:21.251616 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.251394    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/971fec4f-5a07-4eae-b318-0e9c8f381983-kube-api-access-2qn7m\" (UniqueName: \"kubernetes.io/projected/971fec4f-5a07-4eae-b318-0e9c8f381983-kube-api-access-2qn7m\") pod \"971fec4f-5a07-4eae-b318-0e9c8f381983\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") "
Oct 30 19:13:21.251616 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.251449    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-credential-socket" pod "971fec4f-5a07-4eae-b318-0e9c8f381983" (UID: "971fec4f-5a07-4eae-b318-0e9c8f381983"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.251899 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.251676    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-workload-certs" pod "4947eb13-56e8-4d54-b253-21913852aeb1" (UID: "4947eb13-56e8-4d54-b253-21913852aeb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.251961 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.251601    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/971fec4f-5a07-4eae-b318-0e9c8f381983-istiod-ca-cert" pod "971fec4f-5a07-4eae-b318-0e9c8f381983" (UID: "971fec4f-5a07-4eae-b318-0e9c8f381983"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:21.252017 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.251977    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-data\") pod \"971fec4f-5a07-4eae-b318-0e9c8f381983\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") "
Oct 30 19:13:21.252086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.252017    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-podinfo\") pod \"971fec4f-5a07-4eae-b318-0e9c8f381983\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") "
Oct 30 19:13:21.252086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.252056    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-token\" (UniqueName: \"kubernetes.io/projected/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-token\") pod \"971fec4f-5a07-4eae-b318-0e9c8f381983\" (UID: \"971fec4f-5a07-4eae-b318-0e9c8f381983\") "
Oct 30 19:13:21.253098 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.252403    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-data" pod "971fec4f-5a07-4eae-b318-0e9c8f381983" (UID: "971fec4f-5a07-4eae-b318-0e9c8f381983"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.253098 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.252908    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.253098 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.252930    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.253098 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.252947    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.253098 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.252962    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/971fec4f-5a07-4eae-b318-0e9c8f381983-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.253098 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.252976    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.253098 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.252991    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.253098 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.253005    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4947eb13-56e8-4d54-b253-21913852aeb1-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.253098 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.253018    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.253098 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.253031    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.253098 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.253044    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.253787 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.253698    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-istio-envoy" pod "4947eb13-56e8-4d54-b253-21913852aeb1" (UID: "4947eb13-56e8-4d54-b253-21913852aeb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.253787 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.253753    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4947eb13-56e8-4d54-b253-21913852aeb1-istio-token" pod "4947eb13-56e8-4d54-b253-21913852aeb1" (UID: "4947eb13-56e8-4d54-b253-21913852aeb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:21.254210 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.254180    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-envoy" pod "971fec4f-5a07-4eae-b318-0e9c8f381983" (UID: "971fec4f-5a07-4eae-b318-0e9c8f381983"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:21.254811 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.254784    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-token" pod "971fec4f-5a07-4eae-b318-0e9c8f381983" (UID: "971fec4f-5a07-4eae-b318-0e9c8f381983"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:21.254999 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.254982    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4947eb13-56e8-4d54-b253-21913852aeb1-kube-api-access-bvgn5" pod "4947eb13-56e8-4d54-b253-21913852aeb1" (UID: "4947eb13-56e8-4d54-b253-21913852aeb1"). InnerVolumeSpecName "kube-api-access-bvgn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:21.255637 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.255616    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/971fec4f-5a07-4eae-b318-0e9c8f381983-kube-api-access-2qn7m" pod "971fec4f-5a07-4eae-b318-0e9c8f381983" (UID: "971fec4f-5a07-4eae-b318-0e9c8f381983"). InnerVolumeSpecName "kube-api-access-2qn7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:21.256022 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.256006    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-podinfo" pod "971fec4f-5a07-4eae-b318-0e9c8f381983" (UID: "971fec4f-5a07-4eae-b318-0e9c8f381983"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:13:21.257236 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.257215    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4947eb13-56e8-4d54-b253-21913852aeb1-istio-podinfo" pod "4947eb13-56e8-4d54-b253-21913852aeb1" (UID: "4947eb13-56e8-4d54-b253-21913852aeb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:13:21.354970 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.354941    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4947eb13-56e8-4d54-b253-21913852aeb1-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.354970 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.354973    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4947eb13-56e8-4d54-b253-21913852aeb1-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.355208 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.354990    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.355208 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.355004    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvgn5\" (UniqueName: \"kubernetes.io/projected/4947eb13-56e8-4d54-b253-21913852aeb1-kube-api-access-bvgn5\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.355208 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.355017    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4947eb13-56e8-4d54-b253-21913852aeb1-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.355208 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.355030    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qn7m\" (UniqueName: \"kubernetes.io/projected/971fec4f-5a07-4eae-b318-0e9c8f381983-kube-api-access-2qn7m\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.355208 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.355043    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.355208 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.355057    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/971fec4f-5a07-4eae-b318-0e9c8f381983-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:21.808583 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.808496    4924 generic.go:356] "Generic (PLEG): container finished" podID="4947eb13-56e8-4d54-b253-21913852aeb1" containerID="16be12feb68fac8f7a9066d85e368b7d706ffe6d221c3583b9fc937cee169362" exitCode=0
Oct 30 19:13:21.808583 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.808546    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"
Oct 30 19:13:21.808778 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.808582    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl" event={"ID":"4947eb13-56e8-4d54-b253-21913852aeb1","Type":"ContainerDied","Data":"16be12feb68fac8f7a9066d85e368b7d706ffe6d221c3583b9fc937cee169362"}
Oct 30 19:13:21.808778 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.808639    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl" event={"ID":"4947eb13-56e8-4d54-b253-21913852aeb1","Type":"ContainerDied","Data":"08fe2b4acff8d25d7c973148d0f802e2334963dddeabdf1557e131a806ab9d9b"}
Oct 30 19:13:21.808778 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.808669    4924 scope.go:122] "RemoveContainer" containerID="16be12feb68fac8f7a9066d85e368b7d706ffe6d221c3583b9fc937cee169362"
Oct 30 19:13:21.810350 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.810331    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"
Oct 30 19:13:21.810444 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.810361    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5" event={"ID":"0e31d430-f842-4ba1-9890-6b1000521b5d","Type":"ContainerDied","Data":"139bd20044a901cf33b9da08a2ac7193dbea46d720715301adcea54f62b92b59"}
Oct 30 19:13:21.812135 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.812101    4924 generic.go:356] "Generic (PLEG): container finished" podID="971fec4f-5a07-4eae-b318-0e9c8f381983" containerID="57e1d57c7966a5575a5dfe2d548d6e30d4b99a905147c2f995c80e275c47bb63" exitCode=0
Oct 30 19:13:21.812232 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.812165    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"
Oct 30 19:13:21.812232 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.812189    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh" event={"ID":"971fec4f-5a07-4eae-b318-0e9c8f381983","Type":"ContainerDied","Data":"57e1d57c7966a5575a5dfe2d548d6e30d4b99a905147c2f995c80e275c47bb63"}
Oct 30 19:13:21.812232 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.812228    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh" event={"ID":"971fec4f-5a07-4eae-b318-0e9c8f381983","Type":"ContainerDied","Data":"f15eb0ee74d8f5fca6faa0f4c6110c0ebb6dd36991a6bb684ede468bc65266b6"}
Oct 30 19:13:21.820762 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.820737    4924 scope.go:122] "RemoveContainer" containerID="16be12feb68fac8f7a9066d85e368b7d706ffe6d221c3583b9fc937cee169362"
Oct 30 19:13:21.821123 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:13:21.821093    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16be12feb68fac8f7a9066d85e368b7d706ffe6d221c3583b9fc937cee169362\": container with ID starting with 16be12feb68fac8f7a9066d85e368b7d706ffe6d221c3583b9fc937cee169362 not found: ID does not exist" containerID="16be12feb68fac8f7a9066d85e368b7d706ffe6d221c3583b9fc937cee169362"
Oct 30 19:13:21.821200 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.821135    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16be12feb68fac8f7a9066d85e368b7d706ffe6d221c3583b9fc937cee169362"} err="failed to get container status \"16be12feb68fac8f7a9066d85e368b7d706ffe6d221c3583b9fc937cee169362\": rpc error: code = NotFound desc = could not find container \"16be12feb68fac8f7a9066d85e368b7d706ffe6d221c3583b9fc937cee169362\": container with ID starting with 16be12feb68fac8f7a9066d85e368b7d706ffe6d221c3583b9fc937cee169362 not found: ID does not exist"
Oct 30 19:13:21.821200 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.821169    4924 scope.go:122] "RemoveContainer" containerID="d98742fb6ef83fe326b8650726d2dce31df92d65e17f37d4920c5e85d7b4f802"
Oct 30 19:13:21.830910 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.830894    4924 scope.go:122] "RemoveContainer" containerID="57e1d57c7966a5575a5dfe2d548d6e30d4b99a905147c2f995c80e275c47bb63"
Oct 30 19:13:21.840651 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.840634    4924 scope.go:122] "RemoveContainer" containerID="57e1d57c7966a5575a5dfe2d548d6e30d4b99a905147c2f995c80e275c47bb63"
Oct 30 19:13:21.840931 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:13:21.840909    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57e1d57c7966a5575a5dfe2d548d6e30d4b99a905147c2f995c80e275c47bb63\": container with ID starting with 57e1d57c7966a5575a5dfe2d548d6e30d4b99a905147c2f995c80e275c47bb63 not found: ID does not exist" containerID="57e1d57c7966a5575a5dfe2d548d6e30d4b99a905147c2f995c80e275c47bb63"
Oct 30 19:13:21.840999 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:21.840943    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57e1d57c7966a5575a5dfe2d548d6e30d4b99a905147c2f995c80e275c47bb63"} err="failed to get container status \"57e1d57c7966a5575a5dfe2d548d6e30d4b99a905147c2f995c80e275c47bb63\": rpc error: code = NotFound desc = could not find container \"57e1d57c7966a5575a5dfe2d548d6e30d4b99a905147c2f995c80e275c47bb63\": container with ID starting with 57e1d57c7966a5575a5dfe2d548d6e30d4b99a905147c2f995c80e275c47bb63 not found: ID does not exist"
Oct 30 19:13:22.045200 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:22.045163    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"]
Oct 30 19:13:22.070836 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:22.070740    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-tlqvl"]
Oct 30 19:13:22.172934 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:22.172892    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"]
Oct 30 19:13:22.177082 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:22.177047    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-4n2kh"]
Oct 30 19:13:22.281493 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:22.281460    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"]
Oct 30 19:13:22.285034 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:22.285011    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-j84x5"]
Oct 30 19:13:22.620595 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:22.620557    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:22.813575 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:22.813536    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:22.839109 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:22.839046    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e31d430-f842-4ba1-9890-6b1000521b5d" path="/var/lib/kubelet/pods/0e31d430-f842-4ba1-9890-6b1000521b5d/volumes"
Oct 30 19:13:22.839630 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:22.839615    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4947eb13-56e8-4d54-b253-21913852aeb1" path="/var/lib/kubelet/pods/4947eb13-56e8-4d54-b253-21913852aeb1/volumes"
Oct 30 19:13:22.840097 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:22.840083    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="971fec4f-5a07-4eae-b318-0e9c8f381983" path="/var/lib/kubelet/pods/971fec4f-5a07-4eae-b318-0e9c8f381983/volumes"
Oct 30 19:13:25.178088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:25.175608    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:25.376630 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:25.376595    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:25.670879 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:25.670850    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:27.428753 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.428718    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5"]
Oct 30 19:13:27.429616 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.429593    4924 cpu_manager.go:414] "containerMap: removing container" podUID="4947eb13-56e8-4d54-b253-21913852aeb1" containerName="istio-proxy"
Oct 30 19:13:27.429616 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.429616    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4947eb13-56e8-4d54-b253-21913852aeb1" containerName="istio-proxy"
Oct 30 19:13:27.429797 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.429647    4924 cpu_manager.go:414] "containerMap: removing container" podUID="0e31d430-f842-4ba1-9890-6b1000521b5d" containerName="istio-proxy"
Oct 30 19:13:27.429797 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.429655    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e31d430-f842-4ba1-9890-6b1000521b5d" containerName="istio-proxy"
Oct 30 19:13:27.429797 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.429700    4924 cpu_manager.go:414] "containerMap: removing container" podUID="971fec4f-5a07-4eae-b318-0e9c8f381983" containerName="istio-proxy"
Oct 30 19:13:27.429797 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.429707    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="971fec4f-5a07-4eae-b318-0e9c8f381983" containerName="istio-proxy"
Oct 30 19:13:27.429797 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.429724    4924 cpu_manager.go:414] "containerMap: removing container" podUID="36ced340-da28-45e1-867f-de3d1ff4017a" containerName="revoke-keys"
Oct 30 19:13:27.429797 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.429732    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36ced340-da28-45e1-867f-de3d1ff4017a" containerName="revoke-keys"
Oct 30 19:13:27.430149 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.429895    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="971fec4f-5a07-4eae-b318-0e9c8f381983" containerName="istio-proxy"
Oct 30 19:13:27.430149 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.429915    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="36ced340-da28-45e1-867f-de3d1ff4017a" containerName="revoke-keys"
Oct 30 19:13:27.430149 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.429932    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="4947eb13-56e8-4d54-b253-21913852aeb1" containerName="istio-proxy"
Oct 30 19:13:27.430149 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.429945    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e31d430-f842-4ba1-9890-6b1000521b5d" containerName="istio-proxy"
Oct 30 19:13:27.435508 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.435486    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5"
Oct 30 19:13:27.486860 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.486837    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5"]
Oct 30 19:13:27.529835 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.529803    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:13:27.529975 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.529847    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:13:27.529975 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.529889    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:13:27.531563 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.531543    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:13:27.563734 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.563704    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d9cfa7-182c-4f9c-8339-670fdb814216-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5\" (UID: \"02d9cfa7-182c-4f9c-8339-670fdb814216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5"
Oct 30 19:13:27.563901 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.563769    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znl67\" (UniqueName: \"kubernetes.io/projected/02d9cfa7-182c-4f9c-8339-670fdb814216-kube-api-access-znl67\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5\" (UID: \"02d9cfa7-182c-4f9c-8339-670fdb814216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5"
Oct 30 19:13:27.563946 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.563908    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d9cfa7-182c-4f9c-8339-670fdb814216-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5\" (UID: \"02d9cfa7-182c-4f9c-8339-670fdb814216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5"
Oct 30 19:13:27.665811 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.665769    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d9cfa7-182c-4f9c-8339-670fdb814216-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5\" (UID: \"02d9cfa7-182c-4f9c-8339-670fdb814216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5"
Oct 30 19:13:27.666027 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.665876    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znl67\" (UniqueName: \"kubernetes.io/projected/02d9cfa7-182c-4f9c-8339-670fdb814216-kube-api-access-znl67\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5\" (UID: \"02d9cfa7-182c-4f9c-8339-670fdb814216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5"
Oct 30 19:13:27.666027 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.665981    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d9cfa7-182c-4f9c-8339-670fdb814216-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5\" (UID: \"02d9cfa7-182c-4f9c-8339-670fdb814216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5"
Oct 30 19:13:27.666517 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.666491    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d9cfa7-182c-4f9c-8339-670fdb814216-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5\" (UID: \"02d9cfa7-182c-4f9c-8339-670fdb814216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5"
Oct 30 19:13:27.668390 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.668368    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d9cfa7-182c-4f9c-8339-670fdb814216-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5\" (UID: \"02d9cfa7-182c-4f9c-8339-670fdb814216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5"
Oct 30 19:13:27.719896 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.719835    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znl67\" (UniqueName: \"kubernetes.io/projected/02d9cfa7-182c-4f9c-8339-670fdb814216-kube-api-access-znl67\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5\" (UID: \"02d9cfa7-182c-4f9c-8339-670fdb814216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5"
Oct 30 19:13:27.749783 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:27.749762    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5"
Oct 30 19:13:28.127681 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:28.123820    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5"]
Oct 30 19:13:28.131344 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:28.131201    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:28.884345 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:28.884299    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5" event={"ID":"02d9cfa7-182c-4f9c-8339-670fdb814216","Type":"ContainerStarted","Data":"5adc89a054050e72631c3ff58f1a7c7588ce0b9eb03ad3a125551c54604a7ec5"}
Oct 30 19:13:28.884827 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:28.884417    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5" event={"ID":"02d9cfa7-182c-4f9c-8339-670fdb814216","Type":"ContainerStarted","Data":"7c7f2b2fce1012466489c6affd124e8b2b0925261ef4c0afb4625dd8b35086bd"}
Oct 30 19:13:28.998196 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:28.998144    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5" podStartSLOduration=1.9981308819999999 podStartE2EDuration="1.998130882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:28.994952247 +0000 UTC m=+2210.272002693" watchObservedRunningTime="2026-10-30 19:13:28.998130882 +0000 UTC m=+2210.275181342"
Oct 30 19:13:29.893482 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:29.893439    4924 generic.go:356] "Generic (PLEG): container finished" podID="02d9cfa7-182c-4f9c-8339-670fdb814216" containerID="5adc89a054050e72631c3ff58f1a7c7588ce0b9eb03ad3a125551c54604a7ec5" exitCode=0
Oct 30 19:13:29.893928 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:29.893550    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5" event={"ID":"02d9cfa7-182c-4f9c-8339-670fdb814216","Type":"ContainerDied","Data":"5adc89a054050e72631c3ff58f1a7c7588ce0b9eb03ad3a125551c54604a7ec5"}
Oct 30 19:13:31.086384 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:31.086360    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5"
Oct 30 19:13:31.216791 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:31.216695    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02d9cfa7-182c-4f9c-8339-670fdb814216-kube-api-access-znl67\" (UniqueName: \"kubernetes.io/projected/02d9cfa7-182c-4f9c-8339-670fdb814216-kube-api-access-znl67\") pod \"02d9cfa7-182c-4f9c-8339-670fdb814216\" (UID: \"02d9cfa7-182c-4f9c-8339-670fdb814216\") "
Oct 30 19:13:31.216968 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:31.216915    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02d9cfa7-182c-4f9c-8339-670fdb814216-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d9cfa7-182c-4f9c-8339-670fdb814216-maas-api-xks-ca\") pod \"02d9cfa7-182c-4f9c-8339-670fdb814216\" (UID: \"02d9cfa7-182c-4f9c-8339-670fdb814216\") "
Oct 30 19:13:31.217509 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:31.217474    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02d9cfa7-182c-4f9c-8339-670fdb814216-maas-api-openshift-ca" pod "02d9cfa7-182c-4f9c-8339-670fdb814216" (UID: "02d9cfa7-182c-4f9c-8339-670fdb814216"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:31.217631 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:31.217054    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02d9cfa7-182c-4f9c-8339-670fdb814216-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d9cfa7-182c-4f9c-8339-670fdb814216-maas-api-openshift-ca\") pod \"02d9cfa7-182c-4f9c-8339-670fdb814216\" (UID: \"02d9cfa7-182c-4f9c-8339-670fdb814216\") "
Oct 30 19:13:31.218375 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:31.218347    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02d9cfa7-182c-4f9c-8339-670fdb814216-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:31.219169 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:31.219140    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02d9cfa7-182c-4f9c-8339-670fdb814216-maas-api-xks-ca" pod "02d9cfa7-182c-4f9c-8339-670fdb814216" (UID: "02d9cfa7-182c-4f9c-8339-670fdb814216"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:13:31.219628 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:31.219607    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d9cfa7-182c-4f9c-8339-670fdb814216-kube-api-access-znl67" pod "02d9cfa7-182c-4f9c-8339-670fdb814216" (UID: "02d9cfa7-182c-4f9c-8339-670fdb814216"). InnerVolumeSpecName "kube-api-access-znl67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:31.319925 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:31.319895    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znl67\" (UniqueName: \"kubernetes.io/projected/02d9cfa7-182c-4f9c-8339-670fdb814216-kube-api-access-znl67\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:31.319925 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:31.319924    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02d9cfa7-182c-4f9c-8339-670fdb814216-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:31.913168 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:31.913140    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5"
Oct 30 19:13:31.913371 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:31.913164    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5" event={"ID":"02d9cfa7-182c-4f9c-8339-670fdb814216","Type":"ContainerDied","Data":"7c7f2b2fce1012466489c6affd124e8b2b0925261ef4c0afb4625dd8b35086bd"}
Oct 30 19:13:31.913371 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:31.913219    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c7f2b2fce1012466489c6affd124e8b2b0925261ef4c0afb4625dd8b35086bd"
Oct 30 19:13:33.134343 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:33.134301    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:33.348232 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:33.348182    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:33.649317 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:33.649276    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:38.236620 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:38.236581    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:38.930233 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:38.930199    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc"]
Oct 30 19:13:39.031273 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:39.031244    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr"]
Oct 30 19:13:39.328028 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:39.327883    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:39.511469 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:39.511428    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:39.843031 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:39.843001    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:40.060954 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:40.060913    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:40.245473 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:40.245419    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:40.322232 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:40.320918    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd"]
Oct 30 19:13:40.350094 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:40.332648    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3bd578-78hqd"]
Oct 30 19:13:40.565109 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:40.565005    4924 scope.go:122] "RemoveContainer" containerID="dd568467e6e9f1d8d331fabb3622b7df31f7a6d070f6d00317a15f59dfb480cc"
Oct 30 19:13:40.668660 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:40.668632    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:40.839092 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:40.838976    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ca72b99-2e48-42dd-8a6b-cdcc54e153b6" path="/var/lib/kubelet/pods/9ca72b99-2e48-42dd-8a6b-cdcc54e153b6/volumes"
Oct 30 19:13:40.871307 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:40.871265    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:43.079109 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:43.079007    4924 generic.go:356] "Generic (PLEG): container finished" podID="c01a04de-05cf-4bcc-83d3-a7d51ffc9977" containerID="53aafde06247443f6080f5aa3cbe2add276430dd401b6d7a77c7d23da472b5e0" exitCode=0
Oct 30 19:13:43.079529 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:43.079103    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg" event={"ID":"c01a04de-05cf-4bcc-83d3-a7d51ffc9977","Type":"ContainerDied","Data":"53aafde06247443f6080f5aa3cbe2add276430dd401b6d7a77c7d23da472b5e0"}
Oct 30 19:13:43.079529 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:43.079478    4924 scope.go:122] "RemoveContainer" containerID="53aafde06247443f6080f5aa3cbe2add276430dd401b6d7a77c7d23da472b5e0"
Oct 30 19:13:43.081041 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:43.081015    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:13:43.081146 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:43.081102    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:13:43.081188 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:43.081144    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:13:43.931129 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:43.931090    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc" podUID="9ba79faa-3308-4f5a-b30a-9e569ca5d466" containerName="payload-processing" containerID="cri-o://587edc08df4c8cd2782e089f6caf19e9b50560259c8a8c86b982966549934397" gracePeriod=25
Oct 30 19:13:44.032249 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:44.032177    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr" podUID="3f5c238f-665a-4690-8148-cb92c3dd8580" containerName="payload-pre-processing" containerID="cri-o://47d2e0bd4d755e8259efeb11ad187aba122eeb3a02a0997d007849eff7adf271" gracePeriod=25
Oct 30 19:13:44.088766 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:44.088730    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg" event={"ID":"c01a04de-05cf-4bcc-83d3-a7d51ffc9977","Type":"ContainerStarted","Data":"7a0eae0e04271c49494f654b88f1a5377ef0e49822daeafc2ca00a391bde6510"}
Oct 30 19:13:44.090484 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:44.090462    4924 generic.go:356] "Generic (PLEG): container finished" podID="9ba79faa-3308-4f5a-b30a-9e569ca5d466" containerID="587edc08df4c8cd2782e089f6caf19e9b50560259c8a8c86b982966549934397" exitCode=0
Oct 30 19:13:44.090588 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:44.090530    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc" event={"ID":"9ba79faa-3308-4f5a-b30a-9e569ca5d466","Type":"ContainerDied","Data":"587edc08df4c8cd2782e089f6caf19e9b50560259c8a8c86b982966549934397"}
Oct 30 19:13:44.282970 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:44.282937    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc"
Oct 30 19:13:44.369124 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:44.369097    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr"
Oct 30 19:13:44.379729 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:44.379704    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ba79faa-3308-4f5a-b30a-9e569ca5d466-kube-api-access-4cbj5\" (UniqueName: \"kubernetes.io/projected/9ba79faa-3308-4f5a-b30a-9e569ca5d466-kube-api-access-4cbj5\") pod \"9ba79faa-3308-4f5a-b30a-9e569ca5d466\" (UID: \"9ba79faa-3308-4f5a-b30a-9e569ca5d466\") "
Oct 30 19:13:44.379853 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:44.379758    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ba79faa-3308-4f5a-b30a-9e569ca5d466-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba79faa-3308-4f5a-b30a-9e569ca5d466-plugins-config-volume\") pod \"9ba79faa-3308-4f5a-b30a-9e569ca5d466\" (UID: \"9ba79faa-3308-4f5a-b30a-9e569ca5d466\") "
Oct 30 19:13:44.380139 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:44.380112    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ba79faa-3308-4f5a-b30a-9e569ca5d466-plugins-config-volume" pod "9ba79faa-3308-4f5a-b30a-9e569ca5d466" (UID: "9ba79faa-3308-4f5a-b30a-9e569ca5d466"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:44.381713 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:44.381694    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba79faa-3308-4f5a-b30a-9e569ca5d466-kube-api-access-4cbj5" pod "9ba79faa-3308-4f5a-b30a-9e569ca5d466" (UID: "9ba79faa-3308-4f5a-b30a-9e569ca5d466"). InnerVolumeSpecName "kube-api-access-4cbj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:44.481451 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:44.481413    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f5c238f-665a-4690-8148-cb92c3dd8580-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5c238f-665a-4690-8148-cb92c3dd8580-plugins-config-volume\") pod \"3f5c238f-665a-4690-8148-cb92c3dd8580\" (UID: \"3f5c238f-665a-4690-8148-cb92c3dd8580\") "
Oct 30 19:13:44.481627 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:44.481521    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f5c238f-665a-4690-8148-cb92c3dd8580-kube-api-access-mkdq4\" (UniqueName: \"kubernetes.io/projected/3f5c238f-665a-4690-8148-cb92c3dd8580-kube-api-access-mkdq4\") pod \"3f5c238f-665a-4690-8148-cb92c3dd8580\" (UID: \"3f5c238f-665a-4690-8148-cb92c3dd8580\") "
Oct 30 19:13:44.481879 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:44.481866    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cbj5\" (UniqueName: \"kubernetes.io/projected/9ba79faa-3308-4f5a-b30a-9e569ca5d466-kube-api-access-4cbj5\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:44.481920 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:44.481863    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f5c238f-665a-4690-8148-cb92c3dd8580-plugins-config-volume" pod "3f5c238f-665a-4690-8148-cb92c3dd8580" (UID: "3f5c238f-665a-4690-8148-cb92c3dd8580"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:44.481920 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:44.481881    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba79faa-3308-4f5a-b30a-9e569ca5d466-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:44.483570 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:44.483553    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5c238f-665a-4690-8148-cb92c3dd8580-kube-api-access-mkdq4" pod "3f5c238f-665a-4690-8148-cb92c3dd8580" (UID: "3f5c238f-665a-4690-8148-cb92c3dd8580"). InnerVolumeSpecName "kube-api-access-mkdq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:44.583821 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:44.583740    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5c238f-665a-4690-8148-cb92c3dd8580-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:44.583821 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:44.583773    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkdq4\" (UniqueName: \"kubernetes.io/projected/3f5c238f-665a-4690-8148-cb92c3dd8580-kube-api-access-mkdq4\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:44.867446 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:44.867417    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:13:44.868796 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:44.868772    4924 patch_prober.go:29] interesting pod/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 19:13:44.868902 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:44.868823    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg" podUID="c01a04de-05cf-4bcc-83d3-a7d51ffc9977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 19:13:45.099550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:45.099522    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc"
Oct 30 19:13:45.099550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:45.099541    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc" event={"ID":"9ba79faa-3308-4f5a-b30a-9e569ca5d466","Type":"ContainerDied","Data":"9d52b33c806fda732b6e1a4c82bd3ef10e028de04231c18f5af1dc74563ab1f7"}
Oct 30 19:13:45.100043 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:45.099605    4924 scope.go:122] "RemoveContainer" containerID="587edc08df4c8cd2782e089f6caf19e9b50560259c8a8c86b982966549934397"
Oct 30 19:13:45.101091 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:45.101041    4924 generic.go:356] "Generic (PLEG): container finished" podID="3f5c238f-665a-4690-8148-cb92c3dd8580" containerID="47d2e0bd4d755e8259efeb11ad187aba122eeb3a02a0997d007849eff7adf271" exitCode=0
Oct 30 19:13:45.101210 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:45.101104    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr"
Oct 30 19:13:45.101210 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:45.101094    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr" event={"ID":"3f5c238f-665a-4690-8148-cb92c3dd8580","Type":"ContainerDied","Data":"47d2e0bd4d755e8259efeb11ad187aba122eeb3a02a0997d007849eff7adf271"}
Oct 30 19:13:45.101313 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:45.101244    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr" event={"ID":"3f5c238f-665a-4690-8148-cb92c3dd8580","Type":"ContainerDied","Data":"c564a2a62b1a166db32e2c575a84c29c0cd37b9bb6af53f43556a94c5a2cb1a6"}
Oct 30 19:13:45.138703 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:45.137272    4924 scope.go:122] "RemoveContainer" containerID="47d2e0bd4d755e8259efeb11ad187aba122eeb3a02a0997d007849eff7adf271"
Oct 30 19:13:45.147406 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:45.147389    4924 scope.go:122] "RemoveContainer" containerID="47d2e0bd4d755e8259efeb11ad187aba122eeb3a02a0997d007849eff7adf271"
Oct 30 19:13:45.147674 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:13:45.147652    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d2e0bd4d755e8259efeb11ad187aba122eeb3a02a0997d007849eff7adf271\": container with ID starting with 47d2e0bd4d755e8259efeb11ad187aba122eeb3a02a0997d007849eff7adf271 not found: ID does not exist" containerID="47d2e0bd4d755e8259efeb11ad187aba122eeb3a02a0997d007849eff7adf271"
Oct 30 19:13:45.147748 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:45.147683    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d2e0bd4d755e8259efeb11ad187aba122eeb3a02a0997d007849eff7adf271"} err="failed to get container status \"47d2e0bd4d755e8259efeb11ad187aba122eeb3a02a0997d007849eff7adf271\": rpc error: code = NotFound desc = could not find container \"47d2e0bd4d755e8259efeb11ad187aba122eeb3a02a0997d007849eff7adf271\": container with ID starting with 47d2e0bd4d755e8259efeb11ad187aba122eeb3a02a0997d007849eff7adf271 not found: ID does not exist"
Oct 30 19:13:45.275268 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:45.275232    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc"]
Oct 30 19:13:45.280779 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:45.280754    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-8be536-56dfcb9998-8zkvc"]
Oct 30 19:13:45.327084 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:45.322629    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:45.395955 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:45.395858    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr"]
Oct 30 19:13:45.399685 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:45.399660    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-8be536-65864d6d5f-hp2vr"]
Oct 30 19:13:45.652349 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:45.652261    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:13:45.734179 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:13:45.734144    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:13:45.868075 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:45.868039    4924 patch_prober.go:29] interesting pod/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 19:13:45.868255 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:45.868101    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg" podUID="c01a04de-05cf-4bcc-83d3-a7d51ffc9977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 19:13:46.837730 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:46.837698    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f5c238f-665a-4690-8148-cb92c3dd8580" path="/var/lib/kubelet/pods/3f5c238f-665a-4690-8148-cb92c3dd8580/volumes"
Oct 30 19:13:46.838172 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:46.838155    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ba79faa-3308-4f5a-b30a-9e569ca5d466" path="/var/lib/kubelet/pods/9ba79faa-3308-4f5a-b30a-9e569ca5d466/volumes"
Oct 30 19:13:46.868179 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:46.868150    4924 patch_prober.go:29] interesting pod/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 19:13:46.868313 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:46.868201    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg" podUID="c01a04de-05cf-4bcc-83d3-a7d51ffc9977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 19:13:47.268265 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:47.268234    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:47.500086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:47.500019    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:47.575153 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:47.575048    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"]
Oct 30 19:13:47.575322 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:47.575281    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb" podUID="571509d6-5172-43a0-9bc4-29a0da766ddc" containerName="istio-proxy" containerID="cri-o://66cff2ae317acca264fe4144da68fcfed3a2934673a396b9606070f909ff820c" gracePeriod=30
Oct 30 19:13:47.720035 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:47.719999    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:47.868343 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:47.868307    4924 patch_prober.go:29] interesting pod/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 19:13:47.868839 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:47.868358    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg" podUID="c01a04de-05cf-4bcc-83d3-a7d51ffc9977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 19:13:48.247258 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.247182    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc"]
Oct 30 19:13:48.247860 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.247846    4924 cpu_manager.go:414] "containerMap: removing container" podUID="9ba79faa-3308-4f5a-b30a-9e569ca5d466" containerName="payload-processing"
Oct 30 19:13:48.247911 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.247870    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ba79faa-3308-4f5a-b30a-9e569ca5d466" containerName="payload-processing"
Oct 30 19:13:48.247952 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.247913    4924 cpu_manager.go:414] "containerMap: removing container" podUID="02d9cfa7-182c-4f9c-8339-670fdb814216" containerName="revoke-keys"
Oct 30 19:13:48.247952 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.247919    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02d9cfa7-182c-4f9c-8339-670fdb814216" containerName="revoke-keys"
Oct 30 19:13:48.247952 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.247925    4924 cpu_manager.go:414] "containerMap: removing container" podUID="3f5c238f-665a-4690-8148-cb92c3dd8580" containerName="payload-pre-processing"
Oct 30 19:13:48.247952 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.247931    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f5c238f-665a-4690-8148-cb92c3dd8580" containerName="payload-pre-processing"
Oct 30 19:13:48.248088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.248037    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="02d9cfa7-182c-4f9c-8339-670fdb814216" containerName="revoke-keys"
Oct 30 19:13:48.248088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.248044    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f5c238f-665a-4690-8148-cb92c3dd8580" containerName="payload-pre-processing"
Oct 30 19:13:48.248088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.248055    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ba79faa-3308-4f5a-b30a-9e569ca5d466" containerName="payload-processing"
Oct 30 19:13:48.253088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.253047    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc"
Oct 30 19:13:48.317088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.311394    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:13:48.317088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.312223    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:13:48.317088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.312329    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:13:48.317088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.312487    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:13:48.317088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.316136    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc"]
Oct 30 19:13:48.390444 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.390409    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfqsp\" (UniqueName: \"kubernetes.io/projected/62be35dd-f037-4b33-b11a-d1aaf486af36-kube-api-access-dfqsp\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc\" (UID: \"62be35dd-f037-4b33-b11a-d1aaf486af36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc"
Oct 30 19:13:48.390613 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.390591    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62be35dd-f037-4b33-b11a-d1aaf486af36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc\" (UID: \"62be35dd-f037-4b33-b11a-d1aaf486af36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc"
Oct 30 19:13:48.390658 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.390623    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62be35dd-f037-4b33-b11a-d1aaf486af36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc\" (UID: \"62be35dd-f037-4b33-b11a-d1aaf486af36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc"
Oct 30 19:13:48.491452 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.491417    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62be35dd-f037-4b33-b11a-d1aaf486af36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc\" (UID: \"62be35dd-f037-4b33-b11a-d1aaf486af36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc"
Oct 30 19:13:48.491625 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.491460    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62be35dd-f037-4b33-b11a-d1aaf486af36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc\" (UID: \"62be35dd-f037-4b33-b11a-d1aaf486af36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc"
Oct 30 19:13:48.491625 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.491591    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfqsp\" (UniqueName: \"kubernetes.io/projected/62be35dd-f037-4b33-b11a-d1aaf486af36-kube-api-access-dfqsp\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc\" (UID: \"62be35dd-f037-4b33-b11a-d1aaf486af36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc"
Oct 30 19:13:48.492226 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.492200    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62be35dd-f037-4b33-b11a-d1aaf486af36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc\" (UID: \"62be35dd-f037-4b33-b11a-d1aaf486af36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc"
Oct 30 19:13:48.493859 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.493838    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62be35dd-f037-4b33-b11a-d1aaf486af36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc\" (UID: \"62be35dd-f037-4b33-b11a-d1aaf486af36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc"
Oct 30 19:13:48.546124 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.546014    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfqsp\" (UniqueName: \"kubernetes.io/projected/62be35dd-f037-4b33-b11a-d1aaf486af36-kube-api-access-dfqsp\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc\" (UID: \"62be35dd-f037-4b33-b11a-d1aaf486af36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc"
Oct 30 19:13:48.567179 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.567151    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc"
Oct 30 19:13:48.868087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.868032    4924 patch_prober.go:29] interesting pod/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 30 19:13:48.868243 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.868117    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg" podUID="c01a04de-05cf-4bcc-83d3-a7d51ffc9977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 30 19:13:48.962038 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:48.962011    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc"]
Oct 30 19:13:48.963044 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:13:48.962996    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62be35dd_f037_4b33_b11a_d1aaf486af36.slice/crio-29dc72027fa65d96130939545b672b057ebe40679063ca49ebdf9d8fc90f06d9 WatchSource:0}: Error finding container 29dc72027fa65d96130939545b672b057ebe40679063ca49ebdf9d8fc90f06d9: Status 404 returned error can't find the container with id 29dc72027fa65d96130939545b672b057ebe40679063ca49ebdf9d8fc90f06d9
Oct 30 19:13:49.161534 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:49.161504    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc" event={"ID":"62be35dd-f037-4b33-b11a-d1aaf486af36","Type":"ContainerStarted","Data":"29dc72027fa65d96130939545b672b057ebe40679063ca49ebdf9d8fc90f06d9"}
Oct 30 19:13:49.872229 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:49.872195    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:13:49.873229 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:49.873208    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:13:50.171954 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:50.171877    4924 generic.go:356] "Generic (PLEG): container finished" podID="62be35dd-f037-4b33-b11a-d1aaf486af36" containerID="14ebe99a35906ff46b0ad1d48c65760c56576c2a0963d0522e75ca3fe6bb6b44" exitCode=0
Oct 30 19:13:50.172451 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:50.172147    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc" event={"ID":"62be35dd-f037-4b33-b11a-d1aaf486af36","Type":"ContainerDied","Data":"14ebe99a35906ff46b0ad1d48c65760c56576c2a0963d0522e75ca3fe6bb6b44"}
Oct 30 19:13:50.413976 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:50.413941    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:51.337192 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:51.337166    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc"
Oct 30 19:13:51.430892 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:51.430864    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62be35dd-f037-4b33-b11a-d1aaf486af36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62be35dd-f037-4b33-b11a-d1aaf486af36-maas-api-xks-ca\") pod \"62be35dd-f037-4b33-b11a-d1aaf486af36\" (UID: \"62be35dd-f037-4b33-b11a-d1aaf486af36\") "
Oct 30 19:13:51.431056 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:51.431000    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62be35dd-f037-4b33-b11a-d1aaf486af36-kube-api-access-dfqsp\" (UniqueName: \"kubernetes.io/projected/62be35dd-f037-4b33-b11a-d1aaf486af36-kube-api-access-dfqsp\") pod \"62be35dd-f037-4b33-b11a-d1aaf486af36\" (UID: \"62be35dd-f037-4b33-b11a-d1aaf486af36\") "
Oct 30 19:13:51.431056 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:51.431040    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62be35dd-f037-4b33-b11a-d1aaf486af36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62be35dd-f037-4b33-b11a-d1aaf486af36-maas-api-openshift-ca\") pod \"62be35dd-f037-4b33-b11a-d1aaf486af36\" (UID: \"62be35dd-f037-4b33-b11a-d1aaf486af36\") "
Oct 30 19:13:51.431515 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:51.431487    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62be35dd-f037-4b33-b11a-d1aaf486af36-maas-api-openshift-ca" pod "62be35dd-f037-4b33-b11a-d1aaf486af36" (UID: "62be35dd-f037-4b33-b11a-d1aaf486af36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:51.431639 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:51.431624    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62be35dd-f037-4b33-b11a-d1aaf486af36-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:51.432906 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:51.432887    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62be35dd-f037-4b33-b11a-d1aaf486af36-maas-api-xks-ca" pod "62be35dd-f037-4b33-b11a-d1aaf486af36" (UID: "62be35dd-f037-4b33-b11a-d1aaf486af36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:13:51.433058 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:51.433036    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62be35dd-f037-4b33-b11a-d1aaf486af36-kube-api-access-dfqsp" pod "62be35dd-f037-4b33-b11a-d1aaf486af36" (UID: "62be35dd-f037-4b33-b11a-d1aaf486af36"). InnerVolumeSpecName "kube-api-access-dfqsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:51.532805 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:51.532776    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfqsp\" (UniqueName: \"kubernetes.io/projected/62be35dd-f037-4b33-b11a-d1aaf486af36-kube-api-access-dfqsp\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:51.532805 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:51.532806    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62be35dd-f037-4b33-b11a-d1aaf486af36-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:52.193363 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.193332    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc"
Oct 30 19:13:52.193572 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.193365    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc" event={"ID":"62be35dd-f037-4b33-b11a-d1aaf486af36","Type":"ContainerDied","Data":"29dc72027fa65d96130939545b672b057ebe40679063ca49ebdf9d8fc90f06d9"}
Oct 30 19:13:52.193572 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.193413    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29dc72027fa65d96130939545b672b057ebe40679063ca49ebdf9d8fc90f06d9"
Oct 30 19:13:52.873859 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.873832    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:13:52.948901 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.948870    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/571509d6-5172-43a0-9bc4-29a0da766ddc-kube-api-access-j86hb\" (UniqueName: \"kubernetes.io/projected/571509d6-5172-43a0-9bc4-29a0da766ddc-kube-api-access-j86hb\") pod \"571509d6-5172-43a0-9bc4-29a0da766ddc\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") "
Oct 30 19:13:52.949046 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.948953    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-workload-socket\") pod \"571509d6-5172-43a0-9bc4-29a0da766ddc\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") "
Oct 30 19:13:52.949046 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.949003    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-credential-socket\") pod \"571509d6-5172-43a0-9bc4-29a0da766ddc\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") "
Oct 30 19:13:52.949046 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.949037    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-data\") pod \"571509d6-5172-43a0-9bc4-29a0da766ddc\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") "
Oct 30 19:13:52.949256 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.949153    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-token\" (UniqueName: \"kubernetes.io/projected/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-token\") pod \"571509d6-5172-43a0-9bc4-29a0da766ddc\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") "
Oct 30 19:13:52.949256 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.949177    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-workload-socket" pod "571509d6-5172-43a0-9bc4-29a0da766ddc" (UID: "571509d6-5172-43a0-9bc4-29a0da766ddc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:52.949256 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.949208    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/571509d6-5172-43a0-9bc4-29a0da766ddc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/571509d6-5172-43a0-9bc4-29a0da766ddc-istiod-ca-cert\") pod \"571509d6-5172-43a0-9bc4-29a0da766ddc\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") "
Oct 30 19:13:52.949256 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.949245    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-podinfo\") pod \"571509d6-5172-43a0-9bc4-29a0da766ddc\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") "
Oct 30 19:13:52.949459 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.949305    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-credential-socket" pod "571509d6-5172-43a0-9bc4-29a0da766ddc" (UID: "571509d6-5172-43a0-9bc4-29a0da766ddc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:52.949459 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.949329    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-data" pod "571509d6-5172-43a0-9bc4-29a0da766ddc" (UID: "571509d6-5172-43a0-9bc4-29a0da766ddc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:52.949459 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.949394    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-workload-certs\") pod \"571509d6-5172-43a0-9bc4-29a0da766ddc\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") "
Oct 30 19:13:52.949616 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.949509    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-envoy\") pod \"571509d6-5172-43a0-9bc4-29a0da766ddc\" (UID: \"571509d6-5172-43a0-9bc4-29a0da766ddc\") "
Oct 30 19:13:52.949616 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.949518    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/571509d6-5172-43a0-9bc4-29a0da766ddc-istiod-ca-cert" pod "571509d6-5172-43a0-9bc4-29a0da766ddc" (UID: "571509d6-5172-43a0-9bc4-29a0da766ddc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:13:52.949718 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.949700    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-workload-certs" pod "571509d6-5172-43a0-9bc4-29a0da766ddc" (UID: "571509d6-5172-43a0-9bc4-29a0da766ddc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:52.950083 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.950043    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/571509d6-5172-43a0-9bc4-29a0da766ddc-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:52.950186 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.950098    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:52.950186 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.950113    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:52.950186 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.950122    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:52.950186 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.950132    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:52.951516 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.951491    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/571509d6-5172-43a0-9bc4-29a0da766ddc-kube-api-access-j86hb" pod "571509d6-5172-43a0-9bc4-29a0da766ddc" (UID: "571509d6-5172-43a0-9bc4-29a0da766ddc"). InnerVolumeSpecName "kube-api-access-j86hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:52.951850 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.951825    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-envoy" pod "571509d6-5172-43a0-9bc4-29a0da766ddc" (UID: "571509d6-5172-43a0-9bc4-29a0da766ddc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:13:52.952003 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.951984    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-token" pod "571509d6-5172-43a0-9bc4-29a0da766ddc" (UID: "571509d6-5172-43a0-9bc4-29a0da766ddc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:13:52.952152 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:52.952134    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-podinfo" pod "571509d6-5172-43a0-9bc4-29a0da766ddc" (UID: "571509d6-5172-43a0-9bc4-29a0da766ddc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:13:53.051334 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:53.051305    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:53.051334 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:53.051333    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j86hb\" (UniqueName: \"kubernetes.io/projected/571509d6-5172-43a0-9bc4-29a0da766ddc-kube-api-access-j86hb\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:53.051523 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:53.051344    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:53.051523 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:53.051353    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/571509d6-5172-43a0-9bc4-29a0da766ddc-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:13:53.202580 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:53.202542    4924 generic.go:356] "Generic (PLEG): container finished" podID="571509d6-5172-43a0-9bc4-29a0da766ddc" containerID="66cff2ae317acca264fe4144da68fcfed3a2934673a396b9606070f909ff820c" exitCode=0
Oct 30 19:13:53.202784 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:53.202597    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"
Oct 30 19:13:53.202784 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:53.202634    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb" event={"ID":"571509d6-5172-43a0-9bc4-29a0da766ddc","Type":"ContainerDied","Data":"66cff2ae317acca264fe4144da68fcfed3a2934673a396b9606070f909ff820c"}
Oct 30 19:13:53.202784 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:53.202703    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb" event={"ID":"571509d6-5172-43a0-9bc4-29a0da766ddc","Type":"ContainerDied","Data":"20527200146bc840c7a47625112942713facc9302979b1d53b236ea9c6defb6c"}
Oct 30 19:13:53.202784 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:53.202733    4924 scope.go:122] "RemoveContainer" containerID="66cff2ae317acca264fe4144da68fcfed3a2934673a396b9606070f909ff820c"
Oct 30 19:13:53.216376 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:53.216355    4924 scope.go:122] "RemoveContainer" containerID="0388ab91939968b207508aa89dca1f28c77e8bd0354457e0b3fb0a805aa76d89"
Oct 30 19:13:53.226584 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:53.226561    4924 scope.go:122] "RemoveContainer" containerID="66cff2ae317acca264fe4144da68fcfed3a2934673a396b9606070f909ff820c"
Oct 30 19:13:53.226828 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:13:53.226807    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66cff2ae317acca264fe4144da68fcfed3a2934673a396b9606070f909ff820c\": container with ID starting with 66cff2ae317acca264fe4144da68fcfed3a2934673a396b9606070f909ff820c not found: ID does not exist" containerID="66cff2ae317acca264fe4144da68fcfed3a2934673a396b9606070f909ff820c"
Oct 30 19:13:53.226877 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:53.226836    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66cff2ae317acca264fe4144da68fcfed3a2934673a396b9606070f909ff820c"} err="failed to get container status \"66cff2ae317acca264fe4144da68fcfed3a2934673a396b9606070f909ff820c\": rpc error: code = NotFound desc = could not find container \"66cff2ae317acca264fe4144da68fcfed3a2934673a396b9606070f909ff820c\": container with ID starting with 66cff2ae317acca264fe4144da68fcfed3a2934673a396b9606070f909ff820c not found: ID does not exist"
Oct 30 19:13:53.226877 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:53.226855    4924 scope.go:122] "RemoveContainer" containerID="0388ab91939968b207508aa89dca1f28c77e8bd0354457e0b3fb0a805aa76d89"
Oct 30 19:13:53.227105 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:13:53.227082    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0388ab91939968b207508aa89dca1f28c77e8bd0354457e0b3fb0a805aa76d89\": container with ID starting with 0388ab91939968b207508aa89dca1f28c77e8bd0354457e0b3fb0a805aa76d89 not found: ID does not exist" containerID="0388ab91939968b207508aa89dca1f28c77e8bd0354457e0b3fb0a805aa76d89"
Oct 30 19:13:53.227164 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:53.227114    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0388ab91939968b207508aa89dca1f28c77e8bd0354457e0b3fb0a805aa76d89"} err="failed to get container status \"0388ab91939968b207508aa89dca1f28c77e8bd0354457e0b3fb0a805aa76d89\": rpc error: code = NotFound desc = could not find container \"0388ab91939968b207508aa89dca1f28c77e8bd0354457e0b3fb0a805aa76d89\": container with ID starting with 0388ab91939968b207508aa89dca1f28c77e8bd0354457e0b3fb0a805aa76d89 not found: ID does not exist"
Oct 30 19:13:53.366367 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:53.366334    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"]
Oct 30 19:13:53.413151 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:53.413118    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3bd578-openshift-default-b77c55cbc-474sb"]
Oct 30 19:13:54.837874 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:54.837838    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="571509d6-5172-43a0-9bc4-29a0da766ddc" path="/var/lib/kubelet/pods/571509d6-5172-43a0-9bc4-29a0da766ddc/volumes"
Oct 30 19:13:55.112440 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:55.112406    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:56.786343 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:56.786307    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:57.116683 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:57.116643    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:57.312874 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:57.312838    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:57.337937 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:57.337905    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5"]
Oct 30 19:13:57.384384 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:57.384316    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-8be536-fv8t5"]
Oct 30 19:13:57.805899 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:57.805800    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:13:58.836646 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:58.836613    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02d9cfa7-182c-4f9c-8339-670fdb814216" path="/var/lib/kubelet/pods/02d9cfa7-182c-4f9c-8339-670fdb814216/volumes"
Oct 30 19:13:59.988558 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:13:59.988524    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:00.166581 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:00.166548    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:00.361260 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:00.361229    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:02.307864 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:02.307824    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:02.507523 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:02.507487    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:04.707376 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:04.707345    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:04.889679 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:04.889640    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:05.004112 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.003994    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"]
Oct 30 19:14:05.004287 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.004263    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4" podUID="91ef253e-1870-4b85-ab78-f155131276a9" containerName="istio-proxy" containerID="cri-o://73cba2b107500a8fcae094cb4ad7fe365780c8def1a937d44d168e50e85dc409" gracePeriod=30
Oct 30 19:14:05.144540 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.144475    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:05.625631 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.625597    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn"]
Oct 30 19:14:05.626404 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.626384    4924 cpu_manager.go:414] "containerMap: removing container" podUID="571509d6-5172-43a0-9bc4-29a0da766ddc" containerName="istio-proxy"
Oct 30 19:14:05.626404 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.626403    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="571509d6-5172-43a0-9bc4-29a0da766ddc" containerName="istio-proxy"
Oct 30 19:14:05.626552 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.626431    4924 cpu_manager.go:414] "containerMap: removing container" podUID="571509d6-5172-43a0-9bc4-29a0da766ddc" containerName="istio-proxy"
Oct 30 19:14:05.626552 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.626436    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="571509d6-5172-43a0-9bc4-29a0da766ddc" containerName="istio-proxy"
Oct 30 19:14:05.626552 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.626459    4924 cpu_manager.go:414] "containerMap: removing container" podUID="62be35dd-f037-4b33-b11a-d1aaf486af36" containerName="revoke-keys"
Oct 30 19:14:05.626552 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.626465    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62be35dd-f037-4b33-b11a-d1aaf486af36" containerName="revoke-keys"
Oct 30 19:14:05.626684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.626565    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="62be35dd-f037-4b33-b11a-d1aaf486af36" containerName="revoke-keys"
Oct 30 19:14:05.626684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.626578    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="571509d6-5172-43a0-9bc4-29a0da766ddc" containerName="istio-proxy"
Oct 30 19:14:05.626684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.626588    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="571509d6-5172-43a0-9bc4-29a0da766ddc" containerName="istio-proxy"
Oct 30 19:14:05.630280 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.630263    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn"
Oct 30 19:14:05.680943 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.680744    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:14:05.681185 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.680829    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:14:05.682417 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.682384    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:14:05.687211 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.687046    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn"]
Oct 30 19:14:05.727029 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.727001    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:14:05.728737 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.728708    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6vh9\" (UniqueName: \"kubernetes.io/projected/b2eb17cf-81c2-4099-bd64-203112d724e7-kube-api-access-t6vh9\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn\" (UID: \"b2eb17cf-81c2-4099-bd64-203112d724e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn"
Oct 30 19:14:05.728887 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.728873    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2eb17cf-81c2-4099-bd64-203112d724e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn\" (UID: \"b2eb17cf-81c2-4099-bd64-203112d724e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn"
Oct 30 19:14:05.728948 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.728935    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2eb17cf-81c2-4099-bd64-203112d724e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn\" (UID: \"b2eb17cf-81c2-4099-bd64-203112d724e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn"
Oct 30 19:14:05.868380 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.868348    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6vh9\" (UniqueName: \"kubernetes.io/projected/b2eb17cf-81c2-4099-bd64-203112d724e7-kube-api-access-t6vh9\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn\" (UID: \"b2eb17cf-81c2-4099-bd64-203112d724e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn"
Oct 30 19:14:05.868586 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.868504    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2eb17cf-81c2-4099-bd64-203112d724e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn\" (UID: \"b2eb17cf-81c2-4099-bd64-203112d724e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn"
Oct 30 19:14:05.868737 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.868711    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2eb17cf-81c2-4099-bd64-203112d724e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn\" (UID: \"b2eb17cf-81c2-4099-bd64-203112d724e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn"
Oct 30 19:14:05.869539 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.869499    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2eb17cf-81c2-4099-bd64-203112d724e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn\" (UID: \"b2eb17cf-81c2-4099-bd64-203112d724e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn"
Oct 30 19:14:05.871571 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.871549    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2eb17cf-81c2-4099-bd64-203112d724e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn\" (UID: \"b2eb17cf-81c2-4099-bd64-203112d724e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn"
Oct 30 19:14:05.922716 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.922653    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6vh9\" (UniqueName: \"kubernetes.io/projected/b2eb17cf-81c2-4099-bd64-203112d724e7-kube-api-access-t6vh9\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn\" (UID: \"b2eb17cf-81c2-4099-bd64-203112d724e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn"
Oct 30 19:14:05.943957 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:05.943926    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn"
Oct 30 19:14:06.316820 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:06.316790    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn"]
Oct 30 19:14:06.318010 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:14:06.317983    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2eb17cf_81c2_4099_bd64_203112d724e7.slice/crio-a8de38aeb384726ff222afa7986252f0b75f3b09a9fab8714b50f7dea1ddde5a WatchSource:0}: Error finding container a8de38aeb384726ff222afa7986252f0b75f3b09a9fab8714b50f7dea1ddde5a: Status 404 returned error can't find the container with id a8de38aeb384726ff222afa7986252f0b75f3b09a9fab8714b50f7dea1ddde5a
Oct 30 19:14:07.324271 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:07.324237    4924 generic.go:356] "Generic (PLEG): container finished" podID="b2eb17cf-81c2-4099-bd64-203112d724e7" containerID="ac99f8e41c5232e4f600909278548fd6293b1c1a1573607ccf2539ac94779a6f" exitCode=0
Oct 30 19:14:07.324651 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:07.324318    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn" event={"ID":"b2eb17cf-81c2-4099-bd64-203112d724e7","Type":"ContainerDied","Data":"ac99f8e41c5232e4f600909278548fd6293b1c1a1573607ccf2539ac94779a6f"}
Oct 30 19:14:07.324651 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:07.324372    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn" event={"ID":"b2eb17cf-81c2-4099-bd64-203112d724e7","Type":"ContainerStarted","Data":"a8de38aeb384726ff222afa7986252f0b75f3b09a9fab8714b50f7dea1ddde5a"}
Oct 30 19:14:07.836021 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:07.835984    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:08.481453 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:08.481421    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn"
Oct 30 19:14:08.499188 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:08.499166    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2eb17cf-81c2-4099-bd64-203112d724e7-kube-api-access-t6vh9\" (UniqueName: \"kubernetes.io/projected/b2eb17cf-81c2-4099-bd64-203112d724e7-kube-api-access-t6vh9\") pod \"b2eb17cf-81c2-4099-bd64-203112d724e7\" (UID: \"b2eb17cf-81c2-4099-bd64-203112d724e7\") "
Oct 30 19:14:08.499312 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:08.499273    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2eb17cf-81c2-4099-bd64-203112d724e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2eb17cf-81c2-4099-bd64-203112d724e7-maas-api-openshift-ca\") pod \"b2eb17cf-81c2-4099-bd64-203112d724e7\" (UID: \"b2eb17cf-81c2-4099-bd64-203112d724e7\") "
Oct 30 19:14:08.499312 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:08.499309    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2eb17cf-81c2-4099-bd64-203112d724e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2eb17cf-81c2-4099-bd64-203112d724e7-maas-api-xks-ca\") pod \"b2eb17cf-81c2-4099-bd64-203112d724e7\" (UID: \"b2eb17cf-81c2-4099-bd64-203112d724e7\") "
Oct 30 19:14:08.499611 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:08.499583    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2eb17cf-81c2-4099-bd64-203112d724e7-maas-api-openshift-ca" pod "b2eb17cf-81c2-4099-bd64-203112d724e7" (UID: "b2eb17cf-81c2-4099-bd64-203112d724e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:08.501286 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:08.501256    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2eb17cf-81c2-4099-bd64-203112d724e7-maas-api-xks-ca" pod "b2eb17cf-81c2-4099-bd64-203112d724e7" (UID: "b2eb17cf-81c2-4099-bd64-203112d724e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:08.501286 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:08.501267    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2eb17cf-81c2-4099-bd64-203112d724e7-kube-api-access-t6vh9" pod "b2eb17cf-81c2-4099-bd64-203112d724e7" (UID: "b2eb17cf-81c2-4099-bd64-203112d724e7"). InnerVolumeSpecName "kube-api-access-t6vh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:08.601161 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:08.601128    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2eb17cf-81c2-4099-bd64-203112d724e7-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:08.601161 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:08.601160    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6vh9\" (UniqueName: \"kubernetes.io/projected/b2eb17cf-81c2-4099-bd64-203112d724e7-kube-api-access-t6vh9\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:08.601407 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:08.601174    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2eb17cf-81c2-4099-bd64-203112d724e7-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:09.341056 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:09.341025    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn"
Oct 30 19:14:09.341056 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:09.341053    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn" event={"ID":"b2eb17cf-81c2-4099-bd64-203112d724e7","Type":"ContainerDied","Data":"a8de38aeb384726ff222afa7986252f0b75f3b09a9fab8714b50f7dea1ddde5a"}
Oct 30 19:14:09.341310 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:09.341127    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8de38aeb384726ff222afa7986252f0b75f3b09a9fab8714b50f7dea1ddde5a"
Oct 30 19:14:10.296744 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.296719    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:14:10.328259 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.328219    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-istio-data\") pod \"91ef253e-1870-4b85-ab78-f155131276a9\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") "
Oct 30 19:14:10.328401 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.328276    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-credential-socket\") pod \"91ef253e-1870-4b85-ab78-f155131276a9\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") "
Oct 30 19:14:10.328401 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.328325    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-istio-envoy\") pod \"91ef253e-1870-4b85-ab78-f155131276a9\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") "
Oct 30 19:14:10.328401 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.328361    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91ef253e-1870-4b85-ab78-f155131276a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ef253e-1870-4b85-ab78-f155131276a9-istio-podinfo\") pod \"91ef253e-1870-4b85-ab78-f155131276a9\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") "
Oct 30 19:14:10.328575 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.328453    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-istio-data" pod "91ef253e-1870-4b85-ab78-f155131276a9" (UID: "91ef253e-1870-4b85-ab78-f155131276a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:10.328575 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.328507    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-workload-certs\") pod \"91ef253e-1870-4b85-ab78-f155131276a9\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") "
Oct 30 19:14:10.328684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.328567    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-credential-socket" pod "91ef253e-1870-4b85-ab78-f155131276a9" (UID: "91ef253e-1870-4b85-ab78-f155131276a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:10.328684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.328647    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-workload-socket\") pod \"91ef253e-1870-4b85-ab78-f155131276a9\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") "
Oct 30 19:14:10.328796 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.328712    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91ef253e-1870-4b85-ab78-f155131276a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ef253e-1870-4b85-ab78-f155131276a9-istiod-ca-cert\") pod \"91ef253e-1870-4b85-ab78-f155131276a9\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") "
Oct 30 19:14:10.328796 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.328718    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-workload-certs" pod "91ef253e-1870-4b85-ab78-f155131276a9" (UID: "91ef253e-1870-4b85-ab78-f155131276a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:10.328796 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.328765    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ef253e-1870-4b85-ab78-f155131276a9-kube-api-access-lts6d\" (UniqueName: \"kubernetes.io/projected/91ef253e-1870-4b85-ab78-f155131276a9-kube-api-access-lts6d\") pod \"91ef253e-1870-4b85-ab78-f155131276a9\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") "
Oct 30 19:14:10.328951 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.328803    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ef253e-1870-4b85-ab78-f155131276a9-istio-token\" (UniqueName: \"kubernetes.io/projected/91ef253e-1870-4b85-ab78-f155131276a9-istio-token\") pod \"91ef253e-1870-4b85-ab78-f155131276a9\" (UID: \"91ef253e-1870-4b85-ab78-f155131276a9\") "
Oct 30 19:14:10.328951 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.328860    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-workload-socket" pod "91ef253e-1870-4b85-ab78-f155131276a9" (UID: "91ef253e-1870-4b85-ab78-f155131276a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:10.329054 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.329036    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ef253e-1870-4b85-ab78-f155131276a9-istiod-ca-cert" pod "91ef253e-1870-4b85-ab78-f155131276a9" (UID: "91ef253e-1870-4b85-ab78-f155131276a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:10.329472 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.329439    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:10.329597 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.329478    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:10.329597 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.329494    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ef253e-1870-4b85-ab78-f155131276a9-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:10.329597 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.329507    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:10.329597 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.329521    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:10.330613 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.330585    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-istio-envoy" pod "91ef253e-1870-4b85-ab78-f155131276a9" (UID: "91ef253e-1870-4b85-ab78-f155131276a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:10.330613 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.330602    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91ef253e-1870-4b85-ab78-f155131276a9-istio-podinfo" pod "91ef253e-1870-4b85-ab78-f155131276a9" (UID: "91ef253e-1870-4b85-ab78-f155131276a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:14:10.330803 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.330787    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ef253e-1870-4b85-ab78-f155131276a9-istio-token" pod "91ef253e-1870-4b85-ab78-f155131276a9" (UID: "91ef253e-1870-4b85-ab78-f155131276a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:10.330901 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.330880    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ef253e-1870-4b85-ab78-f155131276a9-kube-api-access-lts6d" pod "91ef253e-1870-4b85-ab78-f155131276a9" (UID: "91ef253e-1870-4b85-ab78-f155131276a9"). InnerVolumeSpecName "kube-api-access-lts6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:10.351032 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.351008    4924 generic.go:356] "Generic (PLEG): container finished" podID="91ef253e-1870-4b85-ab78-f155131276a9" containerID="73cba2b107500a8fcae094cb4ad7fe365780c8def1a937d44d168e50e85dc409" exitCode=0
Oct 30 19:14:10.351161 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.351074    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"
Oct 30 19:14:10.351161 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.351095    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4" event={"ID":"91ef253e-1870-4b85-ab78-f155131276a9","Type":"ContainerDied","Data":"73cba2b107500a8fcae094cb4ad7fe365780c8def1a937d44d168e50e85dc409"}
Oct 30 19:14:10.351260 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.351175    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4" event={"ID":"91ef253e-1870-4b85-ab78-f155131276a9","Type":"ContainerDied","Data":"a53230ca5cc69583514b39fbf3ab3a5cb1dc3ab8621311a4f4885fa7dabd80e9"}
Oct 30 19:14:10.351260 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.351200    4924 scope.go:122] "RemoveContainer" containerID="73cba2b107500a8fcae094cb4ad7fe365780c8def1a937d44d168e50e85dc409"
Oct 30 19:14:10.361858 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.361842    4924 scope.go:122] "RemoveContainer" containerID="c4d0f33f078f056566b5489b4da4e50807261f8bd73b63603ed899d11fcafdea"
Oct 30 19:14:10.372209 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.372191    4924 scope.go:122] "RemoveContainer" containerID="73cba2b107500a8fcae094cb4ad7fe365780c8def1a937d44d168e50e85dc409"
Oct 30 19:14:10.372469 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:14:10.372449    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73cba2b107500a8fcae094cb4ad7fe365780c8def1a937d44d168e50e85dc409\": container with ID starting with 73cba2b107500a8fcae094cb4ad7fe365780c8def1a937d44d168e50e85dc409 not found: ID does not exist" containerID="73cba2b107500a8fcae094cb4ad7fe365780c8def1a937d44d168e50e85dc409"
Oct 30 19:14:10.372538 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.372478    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73cba2b107500a8fcae094cb4ad7fe365780c8def1a937d44d168e50e85dc409"} err="failed to get container status \"73cba2b107500a8fcae094cb4ad7fe365780c8def1a937d44d168e50e85dc409\": rpc error: code = NotFound desc = could not find container \"73cba2b107500a8fcae094cb4ad7fe365780c8def1a937d44d168e50e85dc409\": container with ID starting with 73cba2b107500a8fcae094cb4ad7fe365780c8def1a937d44d168e50e85dc409 not found: ID does not exist"
Oct 30 19:14:10.372538 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.372496    4924 scope.go:122] "RemoveContainer" containerID="c4d0f33f078f056566b5489b4da4e50807261f8bd73b63603ed899d11fcafdea"
Oct 30 19:14:10.372750 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:14:10.372734    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4d0f33f078f056566b5489b4da4e50807261f8bd73b63603ed899d11fcafdea\": container with ID starting with c4d0f33f078f056566b5489b4da4e50807261f8bd73b63603ed899d11fcafdea not found: ID does not exist" containerID="c4d0f33f078f056566b5489b4da4e50807261f8bd73b63603ed899d11fcafdea"
Oct 30 19:14:10.372798 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.372753    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d0f33f078f056566b5489b4da4e50807261f8bd73b63603ed899d11fcafdea"} err="failed to get container status \"c4d0f33f078f056566b5489b4da4e50807261f8bd73b63603ed899d11fcafdea\": rpc error: code = NotFound desc = could not find container \"c4d0f33f078f056566b5489b4da4e50807261f8bd73b63603ed899d11fcafdea\": container with ID starting with c4d0f33f078f056566b5489b4da4e50807261f8bd73b63603ed899d11fcafdea not found: ID does not exist"
Oct 30 19:14:10.430826 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.430801    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ef253e-1870-4b85-ab78-f155131276a9-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:10.430943 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.430829    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ef253e-1870-4b85-ab78-f155131276a9-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:10.430943 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.430843    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lts6d\" (UniqueName: \"kubernetes.io/projected/91ef253e-1870-4b85-ab78-f155131276a9-kube-api-access-lts6d\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:10.430943 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.430858    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ef253e-1870-4b85-ab78-f155131276a9-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:10.564819 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.564736    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"]
Oct 30 19:14:10.568191 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.568168    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-8be536-openshift-default-8ffd866c7-fxxp4"]
Oct 30 19:14:10.840828 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:10.840701    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91ef253e-1870-4b85-ab78-f155131276a9" path="/var/lib/kubelet/pods/91ef253e-1870-4b85-ab78-f155131276a9/volumes"
Oct 30 19:14:13.441583 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:13.441548    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:15.712212 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:14:15.712176    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:14:16.107476 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:16.107444    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-568107-zckj8"]
Oct 30 19:14:16.108119 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:16.108101    4924 cpu_manager.go:414] "containerMap: removing container" podUID="91ef253e-1870-4b85-ab78-f155131276a9" containerName="istio-proxy"
Oct 30 19:14:16.108119 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:16.108119    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91ef253e-1870-4b85-ab78-f155131276a9" containerName="istio-proxy"
Oct 30 19:14:16.108240 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:16.108150    4924 cpu_manager.go:414] "containerMap: removing container" podUID="b2eb17cf-81c2-4099-bd64-203112d724e7" containerName="revoke-keys"
Oct 30 19:14:16.108240 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:16.108155    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2eb17cf-81c2-4099-bd64-203112d724e7" containerName="revoke-keys"
Oct 30 19:14:16.108394 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:16.108379    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="91ef253e-1870-4b85-ab78-f155131276a9" containerName="istio-proxy"
Oct 30 19:14:16.108451 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:16.108399    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="91ef253e-1870-4b85-ab78-f155131276a9" containerName="istio-proxy"
Oct 30 19:14:16.108451 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:16.108418    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2eb17cf-81c2-4099-bd64-203112d724e7" containerName="revoke-keys"
Oct 30 19:14:16.114109 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:16.114088    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-568107-zckj8"
Oct 30 19:14:16.168514 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:16.168484    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-568107-zckj8"]
Oct 30 19:14:16.210045 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:16.210017    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:14:16.210206 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:16.210060    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:14:16.210206 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:16.210142    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:14:16.210718 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:16.210705    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:14:16.214442 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:16.214418    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxq67\" (UniqueName: \"kubernetes.io/projected/24e1089b-d671-45ae-8d52-607a705942cc-kube-api-access-pxq67\") pod \"maas-api-revoke-keys-e2e-inf-a-568107-zckj8\" (UID: \"24e1089b-d671-45ae-8d52-607a705942cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-568107-zckj8"
Oct 30 19:14:16.214610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:16.214591    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24e1089b-d671-45ae-8d52-607a705942cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-568107-zckj8\" (UID: \"24e1089b-d671-45ae-8d52-607a705942cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-568107-zckj8"
Oct 30 19:14:16.214690 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:16.214662    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24e1089b-d671-45ae-8d52-607a705942cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-568107-zckj8\" (UID: \"24e1089b-d671-45ae-8d52-607a705942cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-568107-zckj8"
Oct 30 19:14:16.316003 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:16.315966    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxq67\" (UniqueName: \"kubernetes.io/projected/24e1089b-d671-45ae-8d52-607a705942cc-kube-api-access-pxq67\") pod \"maas-api-revoke-keys-e2e-inf-a-568107-zckj8\" (UID: \"24e1089b-d671-45ae-8d52-607a705942cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-568107-zckj8"
Oct 30 19:14:16.316207 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:16.316173    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24e1089b-d671-45ae-8d52-607a705942cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-568107-zckj8\" (UID: \"24e1089b-d671-45ae-8d52-607a705942cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-568107-zckj8"
Oct 30 19:14:16.316292 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:16.316244    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24e1089b-d671-45ae-8d52-607a705942cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-568107-zckj8\" (UID: \"24e1089b-d671-45ae-8d52-607a705942cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-568107-zckj8"
Oct 30 19:14:16.316820 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:16.316788    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24e1089b-d671-45ae-8d52-607a705942cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-568107-zckj8\" (UID: \"24e1089b-d671-45ae-8d52-607a705942cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-568107-zckj8"
Oct 30 19:14:16.318545 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:16.318522    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24e1089b-d671-45ae-8d52-607a705942cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-568107-zckj8\" (UID: \"24e1089b-d671-45ae-8d52-607a705942cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-568107-zckj8"
Oct 30 19:14:16.376219 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:16.376161    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxq67\" (UniqueName: \"kubernetes.io/projected/24e1089b-d671-45ae-8d52-607a705942cc-kube-api-access-pxq67\") pod \"maas-api-revoke-keys-e2e-inf-a-568107-zckj8\" (UID: \"24e1089b-d671-45ae-8d52-607a705942cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-568107-zckj8"
Oct 30 19:14:16.430967 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:16.430939    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-568107-zckj8"
Oct 30 19:14:16.806914 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:16.806883    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-568107-zckj8"]
Oct 30 19:14:16.808136 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:14:16.808107    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24e1089b_d671_45ae_8d52_607a705942cc.slice/crio-b163bba1599377e0e63e6bdbae5c5a8101b5fb7e5df056f61b2d4013ef5a459e WatchSource:0}: Error finding container b163bba1599377e0e63e6bdbae5c5a8101b5fb7e5df056f61b2d4013ef5a459e: Status 404 returned error can't find the container with id b163bba1599377e0e63e6bdbae5c5a8101b5fb7e5df056f61b2d4013ef5a459e
Oct 30 19:14:17.403602 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:17.403512    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-568107-zckj8" event={"ID":"24e1089b-d671-45ae-8d52-607a705942cc","Type":"ContainerStarted","Data":"f6bfa4ab57a980a2e1cdc13e8cd890ea0f42fed57825dfcde3a9307833301d8b"}
Oct 30 19:14:17.403602 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:17.403572    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-568107-zckj8" event={"ID":"24e1089b-d671-45ae-8d52-607a705942cc","Type":"ContainerStarted","Data":"b163bba1599377e0e63e6bdbae5c5a8101b5fb7e5df056f61b2d4013ef5a459e"}
Oct 30 19:14:17.516040 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:17.515985    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-568107-zckj8" podStartSLOduration=1.515970926 podStartE2EDuration="1.515970926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:17.51265877 +0000 UTC m=+2258.789709215" watchObservedRunningTime="2026-10-30 19:14:17.515970926 +0000 UTC m=+2258.793021401"
Oct 30 19:14:18.416523 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:18.416485    4924 generic.go:356] "Generic (PLEG): container finished" podID="24e1089b-d671-45ae-8d52-607a705942cc" containerID="f6bfa4ab57a980a2e1cdc13e8cd890ea0f42fed57825dfcde3a9307833301d8b" exitCode=0
Oct 30 19:14:18.416965 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:18.416524    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-568107-zckj8" event={"ID":"24e1089b-d671-45ae-8d52-607a705942cc","Type":"ContainerDied","Data":"f6bfa4ab57a980a2e1cdc13e8cd890ea0f42fed57825dfcde3a9307833301d8b"}
Oct 30 19:14:18.717500 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:18.717308    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:18.917435 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:18.917297    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:19.106184 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:19.106151    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:19.572011 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:19.571988    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-568107-zckj8"
Oct 30 19:14:19.669278 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:19.669251    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24e1089b-d671-45ae-8d52-607a705942cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24e1089b-d671-45ae-8d52-607a705942cc-maas-api-openshift-ca\") pod \"24e1089b-d671-45ae-8d52-607a705942cc\" (UID: \"24e1089b-d671-45ae-8d52-607a705942cc\") "
Oct 30 19:14:19.669427 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:19.669290    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24e1089b-d671-45ae-8d52-607a705942cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24e1089b-d671-45ae-8d52-607a705942cc-maas-api-xks-ca\") pod \"24e1089b-d671-45ae-8d52-607a705942cc\" (UID: \"24e1089b-d671-45ae-8d52-607a705942cc\") "
Oct 30 19:14:19.669427 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:19.669315    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24e1089b-d671-45ae-8d52-607a705942cc-kube-api-access-pxq67\" (UniqueName: \"kubernetes.io/projected/24e1089b-d671-45ae-8d52-607a705942cc-kube-api-access-pxq67\") pod \"24e1089b-d671-45ae-8d52-607a705942cc\" (UID: \"24e1089b-d671-45ae-8d52-607a705942cc\") "
Oct 30 19:14:19.669610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:19.669585    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24e1089b-d671-45ae-8d52-607a705942cc-maas-api-openshift-ca" pod "24e1089b-d671-45ae-8d52-607a705942cc" (UID: "24e1089b-d671-45ae-8d52-607a705942cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:19.670025 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:19.670002    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24e1089b-d671-45ae-8d52-607a705942cc-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:19.671378 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:19.671354    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24e1089b-d671-45ae-8d52-607a705942cc-maas-api-xks-ca" pod "24e1089b-d671-45ae-8d52-607a705942cc" (UID: "24e1089b-d671-45ae-8d52-607a705942cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:19.671520 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:19.671503    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e1089b-d671-45ae-8d52-607a705942cc-kube-api-access-pxq67" pod "24e1089b-d671-45ae-8d52-607a705942cc" (UID: "24e1089b-d671-45ae-8d52-607a705942cc"). InnerVolumeSpecName "kube-api-access-pxq67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:19.804533 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:19.804455    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24e1089b-d671-45ae-8d52-607a705942cc-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:19.804533 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:19.804484    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxq67\" (UniqueName: \"kubernetes.io/projected/24e1089b-d671-45ae-8d52-607a705942cc-kube-api-access-pxq67\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:20.433261 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:20.433230    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-568107-zckj8"
Oct 30 19:14:20.433261 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:20.433253    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-568107-zckj8" event={"ID":"24e1089b-d671-45ae-8d52-607a705942cc","Type":"ContainerDied","Data":"b163bba1599377e0e63e6bdbae5c5a8101b5fb7e5df056f61b2d4013ef5a459e"}
Oct 30 19:14:20.433511 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:20.433311    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b163bba1599377e0e63e6bdbae5c5a8101b5fb7e5df056f61b2d4013ef5a459e"
Oct 30 19:14:20.463323 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:20.463294    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:20.650782 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:20.650749    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:23.702393 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:23.702352    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:24.769492 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:24.769462    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99"]
Oct 30 19:14:24.869334 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:24.869299    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l"]
Oct 30 19:14:25.160260 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:25.160212    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:25.353584 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:25.353548    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:25.533966 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:25.533879    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:25.718516 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:25.718481    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:25.932228 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:25.932162    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn"]
Oct 30 19:14:25.935476 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:25.935448    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0f3f2f-pb7nn"]
Oct 30 19:14:26.395103 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:26.395054    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:26.835105 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:26.835016    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2eb17cf-81c2-4099-bd64-203112d724e7" path="/var/lib/kubelet/pods/b2eb17cf-81c2-4099-bd64-203112d724e7/volumes"
Oct 30 19:14:29.770500 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:29.770458    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99" podUID="fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e" containerName="payload-processing" containerID="cri-o://f5900b2635d9bcc16af5f060fb9904a9908bb26b86a03b79048f12c7237d6d98" gracePeriod=25
Oct 30 19:14:29.870183 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:29.870145    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l" podUID="65246d5a-1763-4c36-8744-0af6bec35f76" containerName="payload-pre-processing" containerID="cri-o://643ba46f08edd2644883db388a8a892cae0009d4a86d4cf0ed008bc367cc939b" gracePeriod=25
Oct 30 19:14:30.054512 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.054488    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99"
Oct 30 19:14:30.159937 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.159906    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e-plugins-config-volume\") pod \"fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e\" (UID: \"fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e\") "
Oct 30 19:14:30.160057 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.160028    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e-kube-api-access-d975d\" (UniqueName: \"kubernetes.io/projected/fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e-kube-api-access-d975d\") pod \"fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e\" (UID: \"fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e\") "
Oct 30 19:14:30.160226 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.160203    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e-plugins-config-volume" pod "fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e" (UID: "fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:30.160464 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.160448    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:30.161714 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.161695    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l"
Oct 30 19:14:30.162242 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.162224    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e-kube-api-access-d975d" pod "fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e" (UID: "fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e"). InnerVolumeSpecName "kube-api-access-d975d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:30.262011 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.261979    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65246d5a-1763-4c36-8744-0af6bec35f76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65246d5a-1763-4c36-8744-0af6bec35f76-plugins-config-volume\") pod \"65246d5a-1763-4c36-8744-0af6bec35f76\" (UID: \"65246d5a-1763-4c36-8744-0af6bec35f76\") "
Oct 30 19:14:30.262210 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.262161    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65246d5a-1763-4c36-8744-0af6bec35f76-kube-api-access-rgmzh\" (UniqueName: \"kubernetes.io/projected/65246d5a-1763-4c36-8744-0af6bec35f76-kube-api-access-rgmzh\") pod \"65246d5a-1763-4c36-8744-0af6bec35f76\" (UID: \"65246d5a-1763-4c36-8744-0af6bec35f76\") "
Oct 30 19:14:30.262449 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.262423    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65246d5a-1763-4c36-8744-0af6bec35f76-plugins-config-volume" pod "65246d5a-1763-4c36-8744-0af6bec35f76" (UID: "65246d5a-1763-4c36-8744-0af6bec35f76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:30.262634 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.262612    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d975d\" (UniqueName: \"kubernetes.io/projected/fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e-kube-api-access-d975d\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:30.262634 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.262636    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65246d5a-1763-4c36-8744-0af6bec35f76-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:30.264237 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.264216    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65246d5a-1763-4c36-8744-0af6bec35f76-kube-api-access-rgmzh" pod "65246d5a-1763-4c36-8744-0af6bec35f76" (UID: "65246d5a-1763-4c36-8744-0af6bec35f76"). InnerVolumeSpecName "kube-api-access-rgmzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:30.363840 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.363808    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgmzh\" (UniqueName: \"kubernetes.io/projected/65246d5a-1763-4c36-8744-0af6bec35f76-kube-api-access-rgmzh\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:30.508947 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.508914    4924 generic.go:356] "Generic (PLEG): container finished" podID="fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e" containerID="f5900b2635d9bcc16af5f060fb9904a9908bb26b86a03b79048f12c7237d6d98" exitCode=0
Oct 30 19:14:30.509172 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.508960    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99"
Oct 30 19:14:30.509172 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.509000    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99" event={"ID":"fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e","Type":"ContainerDied","Data":"f5900b2635d9bcc16af5f060fb9904a9908bb26b86a03b79048f12c7237d6d98"}
Oct 30 19:14:30.509172 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.509090    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99" event={"ID":"fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e","Type":"ContainerDied","Data":"506d5a8cf6e99a2337048db1d99fd5fb5439ae88d3a016c2cc85e3ef3b2db962"}
Oct 30 19:14:30.509172 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.509102    4924 scope.go:122] "RemoveContainer" containerID="f5900b2635d9bcc16af5f060fb9904a9908bb26b86a03b79048f12c7237d6d98"
Oct 30 19:14:30.510553 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.510525    4924 generic.go:356] "Generic (PLEG): container finished" podID="65246d5a-1763-4c36-8744-0af6bec35f76" containerID="643ba46f08edd2644883db388a8a892cae0009d4a86d4cf0ed008bc367cc939b" exitCode=0
Oct 30 19:14:30.510664 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.510572    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l"
Oct 30 19:14:30.510742 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.510653    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l" event={"ID":"65246d5a-1763-4c36-8744-0af6bec35f76","Type":"ContainerDied","Data":"643ba46f08edd2644883db388a8a892cae0009d4a86d4cf0ed008bc367cc939b"}
Oct 30 19:14:30.510742 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.510706    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l" event={"ID":"65246d5a-1763-4c36-8744-0af6bec35f76","Type":"ContainerDied","Data":"4d7ce13dc64d80dfa2f419e8fc613e901a466abffd9c294edf40f40a261ed569"}
Oct 30 19:14:30.520581 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.520559    4924 scope.go:122] "RemoveContainer" containerID="f5900b2635d9bcc16af5f060fb9904a9908bb26b86a03b79048f12c7237d6d98"
Oct 30 19:14:30.520877 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:14:30.520851    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5900b2635d9bcc16af5f060fb9904a9908bb26b86a03b79048f12c7237d6d98\": container with ID starting with f5900b2635d9bcc16af5f060fb9904a9908bb26b86a03b79048f12c7237d6d98 not found: ID does not exist" containerID="f5900b2635d9bcc16af5f060fb9904a9908bb26b86a03b79048f12c7237d6d98"
Oct 30 19:14:30.520994 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.520879    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5900b2635d9bcc16af5f060fb9904a9908bb26b86a03b79048f12c7237d6d98"} err="failed to get container status \"f5900b2635d9bcc16af5f060fb9904a9908bb26b86a03b79048f12c7237d6d98\": rpc error: code = NotFound desc = could not find container \"f5900b2635d9bcc16af5f060fb9904a9908bb26b86a03b79048f12c7237d6d98\": container with ID starting with f5900b2635d9bcc16af5f060fb9904a9908bb26b86a03b79048f12c7237d6d98 not found: ID does not exist"
Oct 30 19:14:30.520994 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.520897    4924 scope.go:122] "RemoveContainer" containerID="643ba46f08edd2644883db388a8a892cae0009d4a86d4cf0ed008bc367cc939b"
Oct 30 19:14:30.533339 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.533317    4924 scope.go:122] "RemoveContainer" containerID="643ba46f08edd2644883db388a8a892cae0009d4a86d4cf0ed008bc367cc939b"
Oct 30 19:14:30.533699 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:14:30.533678    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"643ba46f08edd2644883db388a8a892cae0009d4a86d4cf0ed008bc367cc939b\": container with ID starting with 643ba46f08edd2644883db388a8a892cae0009d4a86d4cf0ed008bc367cc939b not found: ID does not exist" containerID="643ba46f08edd2644883db388a8a892cae0009d4a86d4cf0ed008bc367cc939b"
Oct 30 19:14:30.533783 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.533709    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"643ba46f08edd2644883db388a8a892cae0009d4a86d4cf0ed008bc367cc939b"} err="failed to get container status \"643ba46f08edd2644883db388a8a892cae0009d4a86d4cf0ed008bc367cc939b\": rpc error: code = NotFound desc = could not find container \"643ba46f08edd2644883db388a8a892cae0009d4a86d4cf0ed008bc367cc939b\": container with ID starting with 643ba46f08edd2644883db388a8a892cae0009d4a86d4cf0ed008bc367cc939b not found: ID does not exist"
Oct 30 19:14:30.666610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.666521    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99"]
Oct 30 19:14:30.670380 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.670353    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-568107-655bd4cbcd-g6l99"]
Oct 30 19:14:30.772916 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.772881    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l"]
Oct 30 19:14:30.776714 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.776689    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-568107-7c8f45b5d7-j254l"]
Oct 30 19:14:30.836394 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.836368    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65246d5a-1763-4c36-8744-0af6bec35f76" path="/var/lib/kubelet/pods/65246d5a-1763-4c36-8744-0af6bec35f76/volumes"
Oct 30 19:14:30.836817 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.836802    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e" path="/var/lib/kubelet/pods/fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e/volumes"
Oct 30 19:14:30.982225 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:30.982135    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:33.387576 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.387544    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"]
Oct 30 19:14:33.388297 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.388279    4924 cpu_manager.go:414] "containerMap: removing container" podUID="24e1089b-d671-45ae-8d52-607a705942cc" containerName="revoke-keys"
Oct 30 19:14:33.388352 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.388299    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24e1089b-d671-45ae-8d52-607a705942cc" containerName="revoke-keys"
Oct 30 19:14:33.388352 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.388314    4924 cpu_manager.go:414] "containerMap: removing container" podUID="fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e" containerName="payload-processing"
Oct 30 19:14:33.388352 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.388319    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e" containerName="payload-processing"
Oct 30 19:14:33.388352 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.388347    4924 cpu_manager.go:414] "containerMap: removing container" podUID="91ef253e-1870-4b85-ab78-f155131276a9" containerName="istio-proxy"
Oct 30 19:14:33.388352 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.388353    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91ef253e-1870-4b85-ab78-f155131276a9" containerName="istio-proxy"
Oct 30 19:14:33.388528 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.388360    4924 cpu_manager.go:414] "containerMap: removing container" podUID="65246d5a-1763-4c36-8744-0af6bec35f76" containerName="payload-pre-processing"
Oct 30 19:14:33.388528 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.388365    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65246d5a-1763-4c36-8744-0af6bec35f76" containerName="payload-pre-processing"
Oct 30 19:14:33.388528 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.388460    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="24e1089b-d671-45ae-8d52-607a705942cc" containerName="revoke-keys"
Oct 30 19:14:33.388528 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.388472    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb2b4a3a-d1c2-48f0-b451-f9b77b3d760e" containerName="payload-processing"
Oct 30 19:14:33.388528 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.388483    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="65246d5a-1763-4c36-8744-0af6bec35f76" containerName="payload-pre-processing"
Oct 30 19:14:33.394905 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.394887    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.445258 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.445234    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-314bd6fa-openshift-default-dockercfg-rtl22\""
Oct 30 19:14:33.495637 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.495606    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"]
Oct 30 19:14:33.504213 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.504188    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-token\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.504383 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.504245    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-workload-certs\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.504383 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.504282    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-workload-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.504383 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.504346    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-podinfo\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.504558 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.504405    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-envoy\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.504558 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.504487    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-credential-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.504719 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.504553    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8dfj\" (UniqueName: \"kubernetes.io/projected/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-kube-api-access-v8dfj\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.504719 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.504589    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-data\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.504825 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.504765    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istiod-ca-cert\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.606112 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.606053    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istiod-ca-cert\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.606331 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.606196    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-token\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.606331 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.606254    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-workload-certs\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.606331 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.606289    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-workload-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.606331 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.606320    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-podinfo\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.606549 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.606449    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-envoy\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.606549 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.606511    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-credential-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.606668 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.606575    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8dfj\" (UniqueName: \"kubernetes.io/projected/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-kube-api-access-v8dfj\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.606668 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.606608    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-data\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.606780 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.606683    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-workload-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.606780 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.606748    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-workload-certs\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.606986 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.606962    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-credential-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.607095 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.607056    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-data\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.607187 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.607159    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istiod-ca-cert\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.608791 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.608761    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-envoy\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.608899 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.608883    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-podinfo\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.665577 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.665520    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8dfj\" (UniqueName: \"kubernetes.io/projected/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-kube-api-access-v8dfj\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.665577 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.665522    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-token\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.713852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.713827    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:33.718635 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.718382    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:33.929137 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:33.929006    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:34.153519 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:34.153489    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"]
Oct 30 19:14:34.153911 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:14:34.153885    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ce2d7c_43f1_41cc_9fb4_2f2e058a4609.slice/crio-97b32ef90853215428b5317e283e004d6d19bbdcf223467a0e3ed06c7db5bad3 WatchSource:0}: Error finding container 97b32ef90853215428b5317e283e004d6d19bbdcf223467a0e3ed06c7db5bad3: Status 404 returned error can't find the container with id 97b32ef90853215428b5317e283e004d6d19bbdcf223467a0e3ed06c7db5bad3
Oct 30 19:14:34.156537 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:34.156507    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:14:34.156602 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:34.156573    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:14:34.156686 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:34.156601    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:14:34.173940 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:34.173915    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:34.365412 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:34.365382    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:34.547048 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:34.546944    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4" event={"ID":"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609","Type":"ContainerStarted","Data":"a0507c3c12405c1dc1082bc99c2fc7b345b63590d6056dbaff0873cb5512c70a"}
Oct 30 19:14:34.547048 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:34.547016    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4" event={"ID":"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609","Type":"ContainerStarted","Data":"97b32ef90853215428b5317e283e004d6d19bbdcf223467a0e3ed06c7db5bad3"}
Oct 30 19:14:34.630234 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:34.630199    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:34.668602 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:34.668535    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4" podStartSLOduration=1.668517652 podStartE2EDuration="1.668517652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:34.661801236 +0000 UTC m=+2275.938851682" watchObservedRunningTime="2026-10-30 19:14:34.668517652 +0000 UTC m=+2275.945568100"
Oct 30 19:14:34.714145 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:34.714108    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:34.876337 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:34.876309    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk"]
Oct 30 19:14:34.887308 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:34.887284    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk"
Oct 30 19:14:34.938842 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:34.938806    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk"]
Oct 30 19:14:34.940387 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:34.940357    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:14:34.940387 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:34.940362    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:14:34.940542 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:34.940363    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:14:34.940542 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:34.940372    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:14:35.038028 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:35.037982    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b17e16-c7c7-4643-be8d-063422f810b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk\" (UID: \"32b17e16-c7c7-4643-be8d-063422f810b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk"
Oct 30 19:14:35.038238 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:35.038045    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9kgp\" (UniqueName: \"kubernetes.io/projected/32b17e16-c7c7-4643-be8d-063422f810b0-kube-api-access-t9kgp\") pod \"maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk\" (UID: \"32b17e16-c7c7-4643-be8d-063422f810b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk"
Oct 30 19:14:35.038238 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:35.038187    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b17e16-c7c7-4643-be8d-063422f810b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk\" (UID: \"32b17e16-c7c7-4643-be8d-063422f810b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk"
Oct 30 19:14:35.140235 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:35.140149    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b17e16-c7c7-4643-be8d-063422f810b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk\" (UID: \"32b17e16-c7c7-4643-be8d-063422f810b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk"
Oct 30 19:14:35.140235 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:35.140193    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9kgp\" (UniqueName: \"kubernetes.io/projected/32b17e16-c7c7-4643-be8d-063422f810b0-kube-api-access-t9kgp\") pod \"maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk\" (UID: \"32b17e16-c7c7-4643-be8d-063422f810b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk"
Oct 30 19:14:35.140446 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:35.140317    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b17e16-c7c7-4643-be8d-063422f810b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk\" (UID: \"32b17e16-c7c7-4643-be8d-063422f810b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk"
Oct 30 19:14:35.140930 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:35.140907    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b17e16-c7c7-4643-be8d-063422f810b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk\" (UID: \"32b17e16-c7c7-4643-be8d-063422f810b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk"
Oct 30 19:14:35.142862 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:35.142835    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b17e16-c7c7-4643-be8d-063422f810b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk\" (UID: \"32b17e16-c7c7-4643-be8d-063422f810b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk"
Oct 30 19:14:35.196865 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:35.196834    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9kgp\" (UniqueName: \"kubernetes.io/projected/32b17e16-c7c7-4643-be8d-063422f810b0-kube-api-access-t9kgp\") pod \"maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk\" (UID: \"32b17e16-c7c7-4643-be8d-063422f810b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk"
Oct 30 19:14:35.231470 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:35.231439    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk"
Oct 30 19:14:35.601635 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:14:35.601603    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b17e16_c7c7_4643_be8d_063422f810b0.slice/crio-aa787f97ae4d09b3950a99a6c5fbf62fdfbae715f854950c5572a8771bc33a16 WatchSource:0}: Error finding container aa787f97ae4d09b3950a99a6c5fbf62fdfbae715f854950c5572a8771bc33a16: Status 404 returned error can't find the container with id aa787f97ae4d09b3950a99a6c5fbf62fdfbae715f854950c5572a8771bc33a16
Oct 30 19:14:35.627108 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:35.601983    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk"]
Oct 30 19:14:35.719088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:35.715812    4924 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 19:14:35.719088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:35.715873    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4" podUID="34ce2d7c-43f1-41cc-9fb4-2f2e058a4609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 19:14:36.066298 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:36.066261    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:36.563711 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:36.563673    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk" event={"ID":"32b17e16-c7c7-4643-be8d-063422f810b0","Type":"ContainerStarted","Data":"7d194b016d0429eea951d4d25a9b618b44963b87e9ed496bf26f2db1a73b6714"}
Oct 30 19:14:36.563889 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:36.563764    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk" event={"ID":"32b17e16-c7c7-4643-be8d-063422f810b0","Type":"ContainerStarted","Data":"aa787f97ae4d09b3950a99a6c5fbf62fdfbae715f854950c5572a8771bc33a16"}
Oct 30 19:14:36.688352 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:36.688291    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk" podStartSLOduration=2.6882767 podStartE2EDuration="2.6882767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:36.686781686 +0000 UTC m=+2277.963832131" watchObservedRunningTime="2026-10-30 19:14:36.6882767 +0000 UTC m=+2277.965327142"
Oct 30 19:14:36.723090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:36.719539    4924 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 19:14:36.723090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:36.719597    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4" podUID="34ce2d7c-43f1-41cc-9fb4-2f2e058a4609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 19:14:37.572674 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:37.572642    4924 generic.go:356] "Generic (PLEG): container finished" podID="32b17e16-c7c7-4643-be8d-063422f810b0" containerID="7d194b016d0429eea951d4d25a9b618b44963b87e9ed496bf26f2db1a73b6714" exitCode=0
Oct 30 19:14:37.572834 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:37.572743    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk" event={"ID":"32b17e16-c7c7-4643-be8d-063422f810b0","Type":"ContainerDied","Data":"7d194b016d0429eea951d4d25a9b618b44963b87e9ed496bf26f2db1a73b6714"}
Oct 30 19:14:37.729484 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:37.729440    4924 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 19:14:37.730095 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:37.730033    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4" podUID="34ce2d7c-43f1-41cc-9fb4-2f2e058a4609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 19:14:38.729038 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:38.729001    4924 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 19:14:38.730367 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:38.730332    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4" podUID="34ce2d7c-43f1-41cc-9fb4-2f2e058a4609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 19:14:38.783254 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:38.783231    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk"
Oct 30 19:14:38.788058 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:38.788034    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32b17e16-c7c7-4643-be8d-063422f810b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b17e16-c7c7-4643-be8d-063422f810b0-maas-api-xks-ca\") pod \"32b17e16-c7c7-4643-be8d-063422f810b0\" (UID: \"32b17e16-c7c7-4643-be8d-063422f810b0\") "
Oct 30 19:14:38.788184 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:38.788142    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32b17e16-c7c7-4643-be8d-063422f810b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b17e16-c7c7-4643-be8d-063422f810b0-maas-api-openshift-ca\") pod \"32b17e16-c7c7-4643-be8d-063422f810b0\" (UID: \"32b17e16-c7c7-4643-be8d-063422f810b0\") "
Oct 30 19:14:38.788248 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:38.788189    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32b17e16-c7c7-4643-be8d-063422f810b0-kube-api-access-t9kgp\" (UniqueName: \"kubernetes.io/projected/32b17e16-c7c7-4643-be8d-063422f810b0-kube-api-access-t9kgp\") pod \"32b17e16-c7c7-4643-be8d-063422f810b0\" (UID: \"32b17e16-c7c7-4643-be8d-063422f810b0\") "
Oct 30 19:14:38.788454 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:38.788431    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32b17e16-c7c7-4643-be8d-063422f810b0-maas-api-openshift-ca" pod "32b17e16-c7c7-4643-be8d-063422f810b0" (UID: "32b17e16-c7c7-4643-be8d-063422f810b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:38.788789 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:38.788770    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32b17e16-c7c7-4643-be8d-063422f810b0-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:38.790032 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:38.790006    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32b17e16-c7c7-4643-be8d-063422f810b0-maas-api-xks-ca" pod "32b17e16-c7c7-4643-be8d-063422f810b0" (UID: "32b17e16-c7c7-4643-be8d-063422f810b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:38.790234 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:38.790214    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b17e16-c7c7-4643-be8d-063422f810b0-kube-api-access-t9kgp" pod "32b17e16-c7c7-4643-be8d-063422f810b0" (UID: "32b17e16-c7c7-4643-be8d-063422f810b0"). InnerVolumeSpecName "kube-api-access-t9kgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:38.890082 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:38.889822    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32b17e16-c7c7-4643-be8d-063422f810b0-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:38.890082 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:38.889853    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9kgp\" (UniqueName: \"kubernetes.io/projected/32b17e16-c7c7-4643-be8d-063422f810b0-kube-api-access-t9kgp\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:39.541740 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:39.541706    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"]
Oct 30 19:14:39.589810 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:39.589785    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk"
Oct 30 19:14:39.589998 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:39.589824    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk" event={"ID":"32b17e16-c7c7-4643-be8d-063422f810b0","Type":"ContainerDied","Data":"aa787f97ae4d09b3950a99a6c5fbf62fdfbae715f854950c5572a8771bc33a16"}
Oct 30 19:14:39.589998 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:39.589912    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa787f97ae4d09b3950a99a6c5fbf62fdfbae715f854950c5572a8771bc33a16"
Oct 30 19:14:39.687646 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:39.687607    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:39.727089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:39.726767    4924 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 19:14:39.727089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:39.726843    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4" podUID="34ce2d7c-43f1-41cc-9fb4-2f2e058a4609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 19:14:40.552303 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:40.552252    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:40.750756 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:40.750721    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:40.771923 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:40.771892    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:40.772443 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:40.772421    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:40.773793 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:40.773776    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:14:41.009379 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:41.009349    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:41.247251 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:41.247210    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:45.771939 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:14:45.771905    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:14:46.625750 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:46.625715    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:48.379967 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.379934    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc"]
Oct 30 19:14:48.541973 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.541945    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"]
Oct 30 19:14:48.542650 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.542632    4924 cpu_manager.go:414] "containerMap: removing container" podUID="32b17e16-c7c7-4643-be8d-063422f810b0" containerName="revoke-keys"
Oct 30 19:14:48.542713 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.542651    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b17e16-c7c7-4643-be8d-063422f810b0" containerName="revoke-keys"
Oct 30 19:14:48.542786 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.542777    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="32b17e16-c7c7-4643-be8d-063422f810b0" containerName="revoke-keys"
Oct 30 19:14:48.548114 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.548090    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"]
Oct 30 19:14:48.548248 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.548162    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.552406 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.552386    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.602555 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.602522    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"]
Oct 30 19:14:48.647836 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.647756    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"]
Oct 30 19:14:48.648290 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.648266    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-workload-certs\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.648417 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.648308    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-credential-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.648476 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.648423    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/555228e6-5a19-4924-a121-9e4d321c0104-istiod-ca-cert\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.648476 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.648453    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghhkq\" (UniqueName: \"kubernetes.io/projected/371f8b40-cf29-431e-be68-d3c7ce0eab02-kube-api-access-ghhkq\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.648558 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.648533    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/555228e6-5a19-4924-a121-9e4d321c0104-istio-token\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.648604 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.648569    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-data\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.648604 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.648597    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371f8b40-cf29-431e-be68-d3c7ce0eab02-istiod-ca-cert\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.648776 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.648707    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-workload-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.648776 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.648757    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-istio-envoy\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.648886 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.648850    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-workload-certs\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.648945 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.648901    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkc2t\" (UniqueName: \"kubernetes.io/projected/555228e6-5a19-4924-a121-9e4d321c0104-kube-api-access-tkc2t\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.648945 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.648934    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-istio-data\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.649057 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.649023    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/555228e6-5a19-4924-a121-9e4d321c0104-istio-podinfo\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.649135 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.649053    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-workload-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.649196 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.649095    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-envoy\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.649326 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.649310    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-podinfo\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.649409 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.649342    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-credential-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.649409 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.649379    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-token\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.751401 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.751365    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-workload-certs\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.751567 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.751410    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkc2t\" (UniqueName: \"kubernetes.io/projected/555228e6-5a19-4924-a121-9e4d321c0104-kube-api-access-tkc2t\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.751567 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.751440    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-istio-data\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.751567 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.751514    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/555228e6-5a19-4924-a121-9e4d321c0104-istio-podinfo\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.751567 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.751542    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-workload-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.751811 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.751567    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-envoy\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.751811 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.751602    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-podinfo\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.751811 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.751633    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-credential-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.751811 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.751667    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-token\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.751811 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.751716    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-workload-certs\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.751811 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.751754    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-credential-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.751811 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.751794    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-workload-certs\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.752222 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.751824    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/555228e6-5a19-4924-a121-9e4d321c0104-istiod-ca-cert\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.752222 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.751856    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghhkq\" (UniqueName: \"kubernetes.io/projected/371f8b40-cf29-431e-be68-d3c7ce0eab02-kube-api-access-ghhkq\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.752222 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.751923    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-istio-data\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.752222 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.751946    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/555228e6-5a19-4924-a121-9e4d321c0104-istio-token\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.752222 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.751991    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-data\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.752222 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.752023    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371f8b40-cf29-431e-be68-d3c7ce0eab02-istiod-ca-cert\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.752222 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.752030    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-credential-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.752222 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.752056    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-workload-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.752222 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.752106    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-workload-certs\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.752222 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.752109    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-istio-envoy\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.752875 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.752733    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-workload-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.752875 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.752787    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-workload-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.752997 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.752938    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/555228e6-5a19-4924-a121-9e4d321c0104-istiod-ca-cert\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.752997 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.752939    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371f8b40-cf29-431e-be68-d3c7ce0eab02-istiod-ca-cert\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.753124 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.753053    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-credential-socket\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.753363 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.753339    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-data\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.754379 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.754357    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-istio-envoy\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.754657 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.754635    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-podinfo\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.755162 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.755136    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-envoy\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.755300 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.755282    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/555228e6-5a19-4924-a121-9e4d321c0104-istio-podinfo\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.772265 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.772241    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:48.806832 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.806793    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-token\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.807379 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.807349    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/555228e6-5a19-4924-a121-9e4d321c0104-istio-token\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.807588 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.807569    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkc2t\" (UniqueName: \"kubernetes.io/projected/555228e6-5a19-4924-a121-9e4d321c0104-kube-api-access-tkc2t\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.807689 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.807669    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghhkq\" (UniqueName: \"kubernetes.io/projected/371f8b40-cf29-431e-be68-d3c7ce0eab02-kube-api-access-ghhkq\") pod \"e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") " pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.863535 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.863504    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:48.869882 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.869859    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:48.951418 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:48.951248    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:49.322577 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:14:49.322514    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371f8b40_cf29_431e_be68_d3c7ce0eab02.slice/crio-5d87658921e1e64a1d39afa38fd51cceabc5aebe273158332dd23d7f3f1d8642 WatchSource:0}: Error finding container 5d87658921e1e64a1d39afa38fd51cceabc5aebe273158332dd23d7f3f1d8642: Status 404 returned error can't find the container with id 5d87658921e1e64a1d39afa38fd51cceabc5aebe273158332dd23d7f3f1d8642
Oct 30 19:14:49.323346 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:49.323178    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"]
Oct 30 19:14:49.325414 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:14:49.325388    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555228e6_5a19_4924_a121_9e4d321c0104.slice/crio-2a72e771036fc36d0217fc3395d5aa9afab7e6ffff2f8cc449ce808d69a14d8f WatchSource:0}: Error finding container 2a72e771036fc36d0217fc3395d5aa9afab7e6ffff2f8cc449ce808d69a14d8f: Status 404 returned error can't find the container with id 2a72e771036fc36d0217fc3395d5aa9afab7e6ffff2f8cc449ce808d69a14d8f
Oct 30 19:14:49.325533 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:49.325489    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:14:49.325610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:49.325552    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:14:49.325610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:49.325586    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:14:49.327477 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:49.327452    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"]
Oct 30 19:14:49.327991 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:49.327963    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:14:49.328121 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:49.328024    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:14:49.328121 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:49.328110    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:14:49.669733 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:49.669692    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9" event={"ID":"371f8b40-cf29-431e-be68-d3c7ce0eab02","Type":"ContainerStarted","Data":"1818fe2ebe802940fb40d3282c59ab94189fd7cd225e0666a90047646f79485c"}
Oct 30 19:14:49.670244 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:49.669755    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9" event={"ID":"371f8b40-cf29-431e-be68-d3c7ce0eab02","Type":"ContainerStarted","Data":"5d87658921e1e64a1d39afa38fd51cceabc5aebe273158332dd23d7f3f1d8642"}
Oct 30 19:14:49.671912 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:49.671884    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw" event={"ID":"555228e6-5a19-4924-a121-9e4d321c0104","Type":"ContainerStarted","Data":"fc019e14612d4637ca7db4f3e772ec5ee0981722a41da1979884e155d5c85e6e"}
Oct 30 19:14:49.672045 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:49.671935    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw" event={"ID":"555228e6-5a19-4924-a121-9e4d321c0104","Type":"ContainerStarted","Data":"2a72e771036fc36d0217fc3395d5aa9afab7e6ffff2f8cc449ce808d69a14d8f"}
Oct 30 19:14:49.790200 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:49.790092    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9" podStartSLOduration=1.7900543899999999 podStartE2EDuration="1.79005439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:49.781503901 +0000 UTC m=+2291.058554347" watchObservedRunningTime="2026-10-30 19:14:49.79005439 +0000 UTC m=+2291.067104844"
Oct 30 19:14:49.871226 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:49.871196    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:49.871866 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:49.871812    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:50.865742 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:50.865709    4924 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:14:50.866282 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:50.865767    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw" podUID="555228e6-5a19-4924-a121-9e4d321c0104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 30 19:14:50.872865 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:50.872839    4924 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 30 19:14:50.872997 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:50.872882    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9" podUID="371f8b40-cf29-431e-be68-d3c7ce0eab02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 30 19:14:51.864006 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:51.863971    4924 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:14:51.864229 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:51.864020    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw" podUID="555228e6-5a19-4924-a121-9e4d321c0104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 30 19:14:51.871368 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:51.871344    4924 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 30 19:14:51.871683 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:51.871379    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9" podUID="371f8b40-cf29-431e-be68-d3c7ce0eab02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 30 19:14:51.872936 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:51.872917    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:52.864878 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:52.864849    4924 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:14:52.865134 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:52.865108    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw" podUID="555228e6-5a19-4924-a121-9e4d321c0104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 30 19:14:52.871191 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:52.871167    4924 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 30 19:14:52.871294 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:52.871203    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9" podUID="371f8b40-cf29-431e-be68-d3c7ce0eab02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 30 19:14:53.449794 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:53.449751    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc" podUID="0b676034-384c-4c43-a8f5-991977f109ce" containerName="payload-processing" containerID="cri-o://83824a1dbc82e10702511263b7f2b315f237370d8490afc2746a66aa7a51ffb2" gracePeriod=25
Oct 30 19:14:53.745145 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:53.744520    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw" podStartSLOduration=5.744497319 podStartE2EDuration="5.744497319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:49.906531684 +0000 UTC m=+2291.183582130" watchObservedRunningTime="2026-10-30 19:14:53.744497319 +0000 UTC m=+2295.021547765"
Oct 30 19:14:53.745145 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:53.744877    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x"]
Oct 30 19:14:53.758864 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:53.758837    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x"
Oct 30 19:14:53.783648 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:53.783616    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x"]
Oct 30 19:14:53.847476 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:53.847440    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:14:53.850488 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:53.847803    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:14:53.850644 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:53.850423    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:14:53.864526 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:53.864498    4924 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:14:53.864730 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:53.864708    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw" podUID="555228e6-5a19-4924-a121-9e4d321c0104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 30 19:14:53.870476 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:53.870445    4924 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 30 19:14:53.870608 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:53.870485    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9" podUID="371f8b40-cf29-431e-be68-d3c7ce0eab02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 30 19:14:53.883392 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:53.883364    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04a304ec-392b-4c28-bb4a-f20d79e1dbee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x\" (UID: \"04a304ec-392b-4c28-bb4a-f20d79e1dbee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x"
Oct 30 19:14:53.883556 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:53.883517    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04a304ec-392b-4c28-bb4a-f20d79e1dbee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x\" (UID: \"04a304ec-392b-4c28-bb4a-f20d79e1dbee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x"
Oct 30 19:14:53.883826 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:53.883808    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9nqb\" (UniqueName: \"kubernetes.io/projected/04a304ec-392b-4c28-bb4a-f20d79e1dbee-kube-api-access-x9nqb\") pod \"maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x\" (UID: \"04a304ec-392b-4c28-bb4a-f20d79e1dbee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x"
Oct 30 19:14:53.894011 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:53.893987    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:14:53.985111 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:53.985051    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9nqb\" (UniqueName: \"kubernetes.io/projected/04a304ec-392b-4c28-bb4a-f20d79e1dbee-kube-api-access-x9nqb\") pod \"maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x\" (UID: \"04a304ec-392b-4c28-bb4a-f20d79e1dbee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x"
Oct 30 19:14:53.985295 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:53.985145    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04a304ec-392b-4c28-bb4a-f20d79e1dbee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x\" (UID: \"04a304ec-392b-4c28-bb4a-f20d79e1dbee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x"
Oct 30 19:14:53.985295 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:53.985205    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04a304ec-392b-4c28-bb4a-f20d79e1dbee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x\" (UID: \"04a304ec-392b-4c28-bb4a-f20d79e1dbee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x"
Oct 30 19:14:53.985886 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:53.985864    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04a304ec-392b-4c28-bb4a-f20d79e1dbee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x\" (UID: \"04a304ec-392b-4c28-bb4a-f20d79e1dbee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x"
Oct 30 19:14:53.988037 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:53.988017    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04a304ec-392b-4c28-bb4a-f20d79e1dbee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x\" (UID: \"04a304ec-392b-4c28-bb4a-f20d79e1dbee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x"
Oct 30 19:14:54.055137 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:54.054490    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9nqb\" (UniqueName: \"kubernetes.io/projected/04a304ec-392b-4c28-bb4a-f20d79e1dbee-kube-api-access-x9nqb\") pod \"maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x\" (UID: \"04a304ec-392b-4c28-bb4a-f20d79e1dbee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x"
Oct 30 19:14:54.079109 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:54.077332    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x"
Oct 30 19:14:54.558032 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:54.557983    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x"]
Oct 30 19:14:54.558604 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:14:54.558194    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a304ec_392b_4c28_bb4a_f20d79e1dbee.slice/crio-3cb1073cd9218969a066135d6f301051ef16f572fd9c5aa552981debd45d7ec8 WatchSource:0}: Error finding container 3cb1073cd9218969a066135d6f301051ef16f572fd9c5aa552981debd45d7ec8: Status 404 returned error can't find the container with id 3cb1073cd9218969a066135d6f301051ef16f572fd9c5aa552981debd45d7ec8
Oct 30 19:14:54.565659 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:54.565625    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb" podUID="3cc610a7-5f70-487c-ba64-452ee8458442" containerName="main" containerID="cri-o://8fd8a489323b20e17dab47bfcb67a0b224a1cdcc33723a5e743381e23eb52ea5" gracePeriod=45
Oct 30 19:14:54.771473 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:54.769297    4924 generic.go:356] "Generic (PLEG): container finished" podID="3cc610a7-5f70-487c-ba64-452ee8458442" containerID="8fd8a489323b20e17dab47bfcb67a0b224a1cdcc33723a5e743381e23eb52ea5" exitCode=0
Oct 30 19:14:54.771473 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:54.769525    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb" event={"ID":"3cc610a7-5f70-487c-ba64-452ee8458442","Type":"ContainerDied","Data":"8fd8a489323b20e17dab47bfcb67a0b224a1cdcc33723a5e743381e23eb52ea5"}
Oct 30 19:14:54.771473 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:54.771247    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x" event={"ID":"04a304ec-392b-4c28-bb4a-f20d79e1dbee","Type":"ContainerStarted","Data":"3cb1073cd9218969a066135d6f301051ef16f572fd9c5aa552981debd45d7ec8"}
Oct 30 19:14:54.872273 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:54.871439    4924 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 30 19:14:54.872273 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:54.871499    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9" podUID="371f8b40-cf29-431e-be68-d3c7ce0eab02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 30 19:14:54.872273 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:54.872144    4924 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:14:54.872273 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:54.872196    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw" podUID="555228e6-5a19-4924-a121-9e4d321c0104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 30 19:14:55.063197 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.058271    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"
Oct 30 19:14:55.157720 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.157678    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cc610a7-5f70-487c-ba64-452ee8458442-tls-certs\" (UniqueName: \"kubernetes.io/secret/3cc610a7-5f70-487c-ba64-452ee8458442-tls-certs\") pod \"3cc610a7-5f70-487c-ba64-452ee8458442\" (UID: \"3cc610a7-5f70-487c-ba64-452ee8458442\") "
Oct 30 19:14:55.157977 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.157957    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-tmp-dir\") pod \"3cc610a7-5f70-487c-ba64-452ee8458442\" (UID: \"3cc610a7-5f70-487c-ba64-452ee8458442\") "
Oct 30 19:14:55.158262 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.158242    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-dshm\") pod \"3cc610a7-5f70-487c-ba64-452ee8458442\" (UID: \"3cc610a7-5f70-487c-ba64-452ee8458442\") "
Oct 30 19:14:55.158448 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.158401    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cc610a7-5f70-487c-ba64-452ee8458442-kube-api-access-2btxd\" (UniqueName: \"kubernetes.io/projected/3cc610a7-5f70-487c-ba64-452ee8458442-kube-api-access-2btxd\") pod \"3cc610a7-5f70-487c-ba64-452ee8458442\" (UID: \"3cc610a7-5f70-487c-ba64-452ee8458442\") "
Oct 30 19:14:55.158670 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.158654    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-home\" (UniqueName: \"kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-home\") pod \"3cc610a7-5f70-487c-ba64-452ee8458442\" (UID: \"3cc610a7-5f70-487c-ba64-452ee8458442\") "
Oct 30 19:14:55.158841 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.158826    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-model-cache\") pod \"3cc610a7-5f70-487c-ba64-452ee8458442\" (UID: \"3cc610a7-5f70-487c-ba64-452ee8458442\") "
Oct 30 19:14:55.159885 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.158287    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-tmp-dir" pod "3cc610a7-5f70-487c-ba64-452ee8458442" (UID: "3cc610a7-5f70-487c-ba64-452ee8458442"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:55.160014 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.159861    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-model-cache" pod "3cc610a7-5f70-487c-ba64-452ee8458442" (UID: "3cc610a7-5f70-487c-ba64-452ee8458442"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:55.160893 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.160258    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-home" pod "3cc610a7-5f70-487c-ba64-452ee8458442" (UID: "3cc610a7-5f70-487c-ba64-452ee8458442"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:55.160893 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.160518    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cc610a7-5f70-487c-ba64-452ee8458442-tls-certs" pod "3cc610a7-5f70-487c-ba64-452ee8458442" (UID: "3cc610a7-5f70-487c-ba64-452ee8458442"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:55.160893 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.160700    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-dshm" pod "3cc610a7-5f70-487c-ba64-452ee8458442" (UID: "3cc610a7-5f70-487c-ba64-452ee8458442"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:55.162892 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.162867    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cc610a7-5f70-487c-ba64-452ee8458442-kube-api-access-2btxd" pod "3cc610a7-5f70-487c-ba64-452ee8458442" (UID: "3cc610a7-5f70-487c-ba64-452ee8458442"). InnerVolumeSpecName "kube-api-access-2btxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:55.273095 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.269286    4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-model-cache\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.273095 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.269326    4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cc610a7-5f70-487c-ba64-452ee8458442-tls-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.273095 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.269341    4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-tmp-dir\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.273095 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.269355    4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-dshm\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.273095 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.269370    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2btxd\" (UniqueName: \"kubernetes.io/projected/3cc610a7-5f70-487c-ba64-452ee8458442-kube-api-access-2btxd\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.273095 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.269387    4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cc610a7-5f70-487c-ba64-452ee8458442-home\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:55.644086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.643512    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:55.786595 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.786555    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x" event={"ID":"04a304ec-392b-4c28-bb4a-f20d79e1dbee","Type":"ContainerStarted","Data":"83390c9f2aad38007fe670e0fa3ee6b64baee1b78c951cb73e48bf62e0164ae7"}
Oct 30 19:14:55.788241 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.788212    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"
Oct 30 19:14:55.788396 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.788240    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb" event={"ID":"3cc610a7-5f70-487c-ba64-452ee8458442","Type":"ContainerDied","Data":"603890db6f4bf913447366612aff886fdfbf0e7be0de7adaf525617b29d73cc4"}
Oct 30 19:14:55.788396 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.788288    4924 scope.go:122] "RemoveContainer" containerID="8fd8a489323b20e17dab47bfcb67a0b224a1cdcc33723a5e743381e23eb52ea5"
Oct 30 19:14:55.830382 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.830344    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:55.864600 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.864571    4924 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Oct 30 19:14:55.864770 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.864624    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw" podUID="555228e6-5a19-4924-a121-9e4d321c0104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Oct 30 19:14:55.870508 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.870461    4924 patch_prober.go:29] interesting pod/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Oct 30 19:14:55.870657 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.870527    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9" podUID="371f8b40-cf29-431e-be68-d3c7ce0eab02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Oct 30 19:14:55.909666 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:55.909571    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x" podStartSLOduration=2.909557613 podStartE2EDuration="2.909557613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:55.905359791 +0000 UTC m=+2297.182410237" watchObservedRunningTime="2026-10-30 19:14:55.909557613 +0000 UTC m=+2297.186608058"
Oct 30 19:14:56.030625 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:56.030560    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"]
Oct 30 19:14:56.035893 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:56.035867    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-649d16/distinct-2-w6-649d16-kserve-64dc8c8668-pb6hb"]
Oct 30 19:14:56.831001 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:56.830966    4924 generic.go:356] "Generic (PLEG): container finished" podID="04a304ec-392b-4c28-bb4a-f20d79e1dbee" containerID="83390c9f2aad38007fe670e0fa3ee6b64baee1b78c951cb73e48bf62e0164ae7" exitCode=0
Oct 30 19:14:56.839140 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:56.839113    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cc610a7-5f70-487c-ba64-452ee8458442" path="/var/lib/kubelet/pods/3cc610a7-5f70-487c-ba64-452ee8458442/volumes"
Oct 30 19:14:56.839481 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:56.839462    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x" event={"ID":"04a304ec-392b-4c28-bb4a-f20d79e1dbee","Type":"ContainerDied","Data":"83390c9f2aad38007fe670e0fa3ee6b64baee1b78c951cb73e48bf62e0164ae7"}
Oct 30 19:14:56.869128 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:56.869104    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:56.869518 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:56.869498    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:56.870286 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:56.870264    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:14:56.875181 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:56.875164    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:56.875444 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:56.875428    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:56.875992 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:56.875975    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:14:57.479854 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:57.477564    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:14:58.064415 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:58.064387    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x"
Oct 30 19:14:58.157890 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:58.157856    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a304ec-392b-4c28-bb4a-f20d79e1dbee-kube-api-access-x9nqb\" (UniqueName: \"kubernetes.io/projected/04a304ec-392b-4c28-bb4a-f20d79e1dbee-kube-api-access-x9nqb\") pod \"04a304ec-392b-4c28-bb4a-f20d79e1dbee\" (UID: \"04a304ec-392b-4c28-bb4a-f20d79e1dbee\") "
Oct 30 19:14:58.158118 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:58.157905    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04a304ec-392b-4c28-bb4a-f20d79e1dbee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04a304ec-392b-4c28-bb4a-f20d79e1dbee-maas-api-openshift-ca\") pod \"04a304ec-392b-4c28-bb4a-f20d79e1dbee\" (UID: \"04a304ec-392b-4c28-bb4a-f20d79e1dbee\") "
Oct 30 19:14:58.158118 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:58.157931    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04a304ec-392b-4c28-bb4a-f20d79e1dbee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04a304ec-392b-4c28-bb4a-f20d79e1dbee-maas-api-xks-ca\") pod \"04a304ec-392b-4c28-bb4a-f20d79e1dbee\" (UID: \"04a304ec-392b-4c28-bb4a-f20d79e1dbee\") "
Oct 30 19:14:58.158350 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:58.158321    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a304ec-392b-4c28-bb4a-f20d79e1dbee-maas-api-openshift-ca" pod "04a304ec-392b-4c28-bb4a-f20d79e1dbee" (UID: "04a304ec-392b-4c28-bb4a-f20d79e1dbee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:14:58.158700 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:58.158673    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04a304ec-392b-4c28-bb4a-f20d79e1dbee-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:58.159962 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:58.159940    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a304ec-392b-4c28-bb4a-f20d79e1dbee-kube-api-access-x9nqb" pod "04a304ec-392b-4c28-bb4a-f20d79e1dbee" (UID: "04a304ec-392b-4c28-bb4a-f20d79e1dbee"). InnerVolumeSpecName "kube-api-access-x9nqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:58.160055 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:58.159994    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04a304ec-392b-4c28-bb4a-f20d79e1dbee-maas-api-xks-ca" pod "04a304ec-392b-4c28-bb4a-f20d79e1dbee" (UID: "04a304ec-392b-4c28-bb4a-f20d79e1dbee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:58.260509 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:58.260478    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9nqb\" (UniqueName: \"kubernetes.io/projected/04a304ec-392b-4c28-bb4a-f20d79e1dbee-kube-api-access-x9nqb\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:58.260509 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:58.260506    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04a304ec-392b-4c28-bb4a-f20d79e1dbee-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:58.850567 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:58.850534    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x"
Oct 30 19:14:58.850700 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:58.850581    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x" event={"ID":"04a304ec-392b-4c28-bb4a-f20d79e1dbee","Type":"ContainerDied","Data":"3cb1073cd9218969a066135d6f301051ef16f572fd9c5aa552981debd45d7ec8"}
Oct 30 19:14:58.850700 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:14:58.850631    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cb1073cd9218969a066135d6f301051ef16f572fd9c5aa552981debd45d7ec8"
Oct 30 19:15:00.168813 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.168766    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4"]
Oct 30 19:15:00.169745 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.169721    4924 cpu_manager.go:414] "containerMap: removing container" podUID="3cc610a7-5f70-487c-ba64-452ee8458442" containerName="main"
Oct 30 19:15:00.169745 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.169744    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cc610a7-5f70-487c-ba64-452ee8458442" containerName="main"
Oct 30 19:15:00.169961 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.169762    4924 cpu_manager.go:414] "containerMap: removing container" podUID="04a304ec-392b-4c28-bb4a-f20d79e1dbee" containerName="revoke-keys"
Oct 30 19:15:00.169961 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.169770    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a304ec-392b-4c28-bb4a-f20d79e1dbee" containerName="revoke-keys"
Oct 30 19:15:00.170092 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.170027    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cc610a7-5f70-487c-ba64-452ee8458442" containerName="main"
Oct 30 19:15:00.170092 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.170046    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a304ec-392b-4c28-bb4a-f20d79e1dbee" containerName="revoke-keys"
Oct 30 19:15:00.173814 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.173793    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg"]
Oct 30 19:15:00.173931 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.173909    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4"
Oct 30 19:15:00.178087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.178041    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q"]
Oct 30 19:15:00.178249 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.178159    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg"
Oct 30 19:15:00.182616 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.182594    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q"
Oct 30 19:15:00.231298 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.231263    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr"]
Oct 30 19:15:00.235755 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.235732    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g"]
Oct 30 19:15:00.235895 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.235838    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr"
Oct 30 19:15:00.239983 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.239958    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh"]
Oct 30 19:15:00.240152 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.240050    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g"
Oct 30 19:15:00.246012 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.245993    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf"]
Oct 30 19:15:00.246357 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.246147    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh"
Oct 30 19:15:00.250782 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.250761    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx"]
Oct 30 19:15:00.250867 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.250849    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf"
Oct 30 19:15:00.255800 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.255781    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96"]
Oct 30 19:15:00.255884 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.255867    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx"
Oct 30 19:15:00.260862 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.260845    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96"
Oct 30 19:15:00.271106 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.271082    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:15:00.271234 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.271118    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:15:00.271234 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.271198    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:15:00.275123 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.275103    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj"]
Oct 30 19:15:00.291684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.291643    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/000b4c54-e96e-497e-b489-504f2f52094c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4\" (UID: \"000b4c54-e96e-497e-b489-504f2f52094c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4"
Oct 30 19:15:00.291868 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.291716    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gvq2\" (UniqueName: \"kubernetes.io/projected/d9d33ee4-cc7c-4043-a179-65094f01ca5f-kube-api-access-8gvq2\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg\" (UID: \"d9d33ee4-cc7c-4043-a179-65094f01ca5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg"
Oct 30 19:15:00.293089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.291863    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/142bd310-9137-4db5-a5e5-27fcf478fbb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889795-bjw2q\" (UID: \"142bd310-9137-4db5-a5e5-27fcf478fbb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q"
Oct 30 19:15:00.293089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.291907    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d33ee4-cc7c-4043-a179-65094f01ca5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg\" (UID: \"d9d33ee4-cc7c-4043-a179-65094f01ca5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg"
Oct 30 19:15:00.293089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.291960    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbhl8\" (UniqueName: \"kubernetes.io/projected/142bd310-9137-4db5-a5e5-27fcf478fbb5-kube-api-access-fbhl8\") pod \"maas-api-key-cleanup-29889795-bjw2q\" (UID: \"142bd310-9137-4db5-a5e5-27fcf478fbb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q"
Oct 30 19:15:00.293089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.292000    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdw2f\" (UniqueName: \"kubernetes.io/projected/000b4c54-e96e-497e-b489-504f2f52094c-kube-api-access-kdw2f\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4\" (UID: \"000b4c54-e96e-497e-b489-504f2f52094c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4"
Oct 30 19:15:00.293089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.292977    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg"]
Oct 30 19:15:00.294033 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.293117    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj"
Oct 30 19:15:00.298498 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.298475    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q"]
Oct 30 19:15:00.361712 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.361676    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4"]
Oct 30 19:15:00.370275 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.370252    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr"]
Oct 30 19:15:00.375924 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.373767    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g"]
Oct 30 19:15:00.377101 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.377019    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh"]
Oct 30 19:15:00.381038 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.381019    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj"]
Oct 30 19:15:00.393609 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.393585    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea62e28-48cf-40e4-9de4-1b767bd839ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96\" (UID: \"4ea62e28-48cf-40e4-9de4-1b767bd839ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96"
Oct 30 19:15:00.393703 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.393633    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nqv9\" (UniqueName: \"kubernetes.io/projected/7f9d64eb-f439-47cc-8f2c-d7eaf03fd573-kube-api-access-5nqv9\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf\" (UID: \"7f9d64eb-f439-47cc-8f2c-d7eaf03fd573\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf"
Oct 30 19:15:00.393842 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.393802    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/142bd310-9137-4db5-a5e5-27fcf478fbb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889795-bjw2q\" (UID: \"142bd310-9137-4db5-a5e5-27fcf478fbb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q"
Oct 30 19:15:00.393913 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.393874    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d33ee4-cc7c-4043-a179-65094f01ca5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg\" (UID: \"d9d33ee4-cc7c-4043-a179-65094f01ca5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg"
Oct 30 19:15:00.393978 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.393934    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70bd8cf2-4a28-46e2-8223-27e0890f7444-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx\" (UID: \"70bd8cf2-4a28-46e2-8223-27e0890f7444\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx"
Oct 30 19:15:00.394032 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.393984    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr\" (UID: \"e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr"
Oct 30 19:15:00.394032 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.394010    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbhl8\" (UniqueName: \"kubernetes.io/projected/142bd310-9137-4db5-a5e5-27fcf478fbb5-kube-api-access-fbhl8\") pod \"maas-api-key-cleanup-29889795-bjw2q\" (UID: \"142bd310-9137-4db5-a5e5-27fcf478fbb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q"
Oct 30 19:15:00.394257 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.394235    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdw2f\" (UniqueName: \"kubernetes.io/projected/000b4c54-e96e-497e-b489-504f2f52094c-kube-api-access-kdw2f\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4\" (UID: \"000b4c54-e96e-497e-b489-504f2f52094c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4"
Oct 30 19:15:00.394328 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.394293    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnng8\" (UniqueName: \"kubernetes.io/projected/4cbfab36-1855-4785-97e2-4c8a96cd9e0c-kube-api-access-bnng8\") pod \"maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj\" (UID: \"4cbfab36-1855-4785-97e2-4c8a96cd9e0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj"
Oct 30 19:15:00.394328 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.394320    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6n9h\" (UniqueName: \"kubernetes.io/projected/e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1-kube-api-access-n6n9h\") pod \"maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr\" (UID: \"e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr"
Oct 30 19:15:00.394446 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.394401    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hj8r\" (UniqueName: \"kubernetes.io/projected/534a63d0-c4b9-48cb-88f0-bed24a366017-kube-api-access-5hj8r\") pod \"maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g\" (UID: \"534a63d0-c4b9-48cb-88f0-bed24a366017\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g"
Oct 30 19:15:00.394508 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.394471    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6c5h\" (UniqueName: \"kubernetes.io/projected/70bd8cf2-4a28-46e2-8223-27e0890f7444-kube-api-access-t6c5h\") pod \"maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx\" (UID: \"70bd8cf2-4a28-46e2-8223-27e0890f7444\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx"
Oct 30 19:15:00.394508 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.394499    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f9d64eb-f439-47cc-8f2c-d7eaf03fd573-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf\" (UID: \"7f9d64eb-f439-47cc-8f2c-d7eaf03fd573\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf"
Oct 30 19:15:00.394634 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.394505    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d33ee4-cc7c-4043-a179-65094f01ca5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg\" (UID: \"d9d33ee4-cc7c-4043-a179-65094f01ca5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg"
Oct 30 19:15:00.394634 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.394570    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhnl5\" (UniqueName: \"kubernetes.io/projected/d9aa4147-99fd-4ee4-9dd0-43ae21407002-kube-api-access-bhnl5\") pod \"maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh\" (UID: \"d9aa4147-99fd-4ee4-9dd0-43ae21407002\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh"
Oct 30 19:15:00.394634 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.394621    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/534a63d0-c4b9-48cb-88f0-bed24a366017-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g\" (UID: \"534a63d0-c4b9-48cb-88f0-bed24a366017\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g"
Oct 30 19:15:00.394787 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.394649    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qps2\" (UniqueName: \"kubernetes.io/projected/4ea62e28-48cf-40e4-9de4-1b767bd839ce-kube-api-access-8qps2\") pod \"maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96\" (UID: \"4ea62e28-48cf-40e4-9de4-1b767bd839ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96"
Oct 30 19:15:00.394787 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.394669    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/142bd310-9137-4db5-a5e5-27fcf478fbb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889795-bjw2q\" (UID: \"142bd310-9137-4db5-a5e5-27fcf478fbb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q"
Oct 30 19:15:00.394787 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.394783    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/000b4c54-e96e-497e-b489-504f2f52094c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4\" (UID: \"000b4c54-e96e-497e-b489-504f2f52094c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4"
Oct 30 19:15:00.394943 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.394871    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cbfab36-1855-4785-97e2-4c8a96cd9e0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj\" (UID: \"4cbfab36-1855-4785-97e2-4c8a96cd9e0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj"
Oct 30 19:15:00.394943 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.394915    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gvq2\" (UniqueName: \"kubernetes.io/projected/d9d33ee4-cc7c-4043-a179-65094f01ca5f-kube-api-access-8gvq2\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg\" (UID: \"d9d33ee4-cc7c-4043-a179-65094f01ca5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg"
Oct 30 19:15:00.395220 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.394951    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9aa4147-99fd-4ee4-9dd0-43ae21407002-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh\" (UID: \"d9aa4147-99fd-4ee4-9dd0-43ae21407002\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh"
Oct 30 19:15:00.395372 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.395355    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/000b4c54-e96e-497e-b489-504f2f52094c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4\" (UID: \"000b4c54-e96e-497e-b489-504f2f52094c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4"
Oct 30 19:15:00.400694 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.400671    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf"]
Oct 30 19:15:00.441471 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.441396    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx"]
Oct 30 19:15:00.448884 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.448812    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96"]
Oct 30 19:15:00.449685 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.449664    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gvq2\" (UniqueName: \"kubernetes.io/projected/d9d33ee4-cc7c-4043-a179-65094f01ca5f-kube-api-access-8gvq2\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg\" (UID: \"d9d33ee4-cc7c-4043-a179-65094f01ca5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg"
Oct 30 19:15:00.449827 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.449712    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdw2f\" (UniqueName: \"kubernetes.io/projected/000b4c54-e96e-497e-b489-504f2f52094c-kube-api-access-kdw2f\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4\" (UID: \"000b4c54-e96e-497e-b489-504f2f52094c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4"
Oct 30 19:15:00.450805 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.450763    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbhl8\" (UniqueName: \"kubernetes.io/projected/142bd310-9137-4db5-a5e5-27fcf478fbb5-kube-api-access-fbhl8\") pod \"maas-api-key-cleanup-29889795-bjw2q\" (UID: \"142bd310-9137-4db5-a5e5-27fcf478fbb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q"
Oct 30 19:15:00.489681 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.489658    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4"
Oct 30 19:15:00.496610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.496588    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6c5h\" (UniqueName: \"kubernetes.io/projected/70bd8cf2-4a28-46e2-8223-27e0890f7444-kube-api-access-t6c5h\") pod \"maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx\" (UID: \"70bd8cf2-4a28-46e2-8223-27e0890f7444\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx"
Oct 30 19:15:00.496685 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.496619    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f9d64eb-f439-47cc-8f2c-d7eaf03fd573-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf\" (UID: \"7f9d64eb-f439-47cc-8f2c-d7eaf03fd573\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf"
Oct 30 19:15:00.496685 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.496647    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhnl5\" (UniqueName: \"kubernetes.io/projected/d9aa4147-99fd-4ee4-9dd0-43ae21407002-kube-api-access-bhnl5\") pod \"maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh\" (UID: \"d9aa4147-99fd-4ee4-9dd0-43ae21407002\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh"
Oct 30 19:15:00.496685 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.496678    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/534a63d0-c4b9-48cb-88f0-bed24a366017-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g\" (UID: \"534a63d0-c4b9-48cb-88f0-bed24a366017\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g"
Oct 30 19:15:00.496843 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.496696    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qps2\" (UniqueName: \"kubernetes.io/projected/4ea62e28-48cf-40e4-9de4-1b767bd839ce-kube-api-access-8qps2\") pod \"maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96\" (UID: \"4ea62e28-48cf-40e4-9de4-1b767bd839ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96"
Oct 30 19:15:00.496843 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.496742    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cbfab36-1855-4785-97e2-4c8a96cd9e0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj\" (UID: \"4cbfab36-1855-4785-97e2-4c8a96cd9e0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj"
Oct 30 19:15:00.496843 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.496767    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9aa4147-99fd-4ee4-9dd0-43ae21407002-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh\" (UID: \"d9aa4147-99fd-4ee4-9dd0-43ae21407002\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh"
Oct 30 19:15:00.496843 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.496806    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea62e28-48cf-40e4-9de4-1b767bd839ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96\" (UID: \"4ea62e28-48cf-40e4-9de4-1b767bd839ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96"
Oct 30 19:15:00.496843 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.496844    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nqv9\" (UniqueName: \"kubernetes.io/projected/7f9d64eb-f439-47cc-8f2c-d7eaf03fd573-kube-api-access-5nqv9\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf\" (UID: \"7f9d64eb-f439-47cc-8f2c-d7eaf03fd573\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf"
Oct 30 19:15:00.497123 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.496897    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70bd8cf2-4a28-46e2-8223-27e0890f7444-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx\" (UID: \"70bd8cf2-4a28-46e2-8223-27e0890f7444\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx"
Oct 30 19:15:00.497123 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.496919    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr\" (UID: \"e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr"
Oct 30 19:15:00.497123 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.496967    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnng8\" (UniqueName: \"kubernetes.io/projected/4cbfab36-1855-4785-97e2-4c8a96cd9e0c-kube-api-access-bnng8\") pod \"maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj\" (UID: \"4cbfab36-1855-4785-97e2-4c8a96cd9e0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj"
Oct 30 19:15:00.497294 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.497113    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6n9h\" (UniqueName: \"kubernetes.io/projected/e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1-kube-api-access-n6n9h\") pod \"maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr\" (UID: \"e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr"
Oct 30 19:15:00.497294 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.497165    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hj8r\" (UniqueName: \"kubernetes.io/projected/534a63d0-c4b9-48cb-88f0-bed24a366017-kube-api-access-5hj8r\") pod \"maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g\" (UID: \"534a63d0-c4b9-48cb-88f0-bed24a366017\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g"
Oct 30 19:15:00.497509 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.497485    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/534a63d0-c4b9-48cb-88f0-bed24a366017-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g\" (UID: \"534a63d0-c4b9-48cb-88f0-bed24a366017\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g"
Oct 30 19:15:00.497584 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.497541    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f9d64eb-f439-47cc-8f2c-d7eaf03fd573-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf\" (UID: \"7f9d64eb-f439-47cc-8f2c-d7eaf03fd573\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf"
Oct 30 19:15:00.497584 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.497554    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cbfab36-1855-4785-97e2-4c8a96cd9e0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj\" (UID: \"4cbfab36-1855-4785-97e2-4c8a96cd9e0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj"
Oct 30 19:15:00.497705 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.497658    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9aa4147-99fd-4ee4-9dd0-43ae21407002-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh\" (UID: \"d9aa4147-99fd-4ee4-9dd0-43ae21407002\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh"
Oct 30 19:15:00.497849 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.497828    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg"
Oct 30 19:15:00.498187 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.498167    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70bd8cf2-4a28-46e2-8223-27e0890f7444-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx\" (UID: \"70bd8cf2-4a28-46e2-8223-27e0890f7444\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx"
Oct 30 19:15:00.498260 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.498201    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea62e28-48cf-40e4-9de4-1b767bd839ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96\" (UID: \"4ea62e28-48cf-40e4-9de4-1b767bd839ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96"
Oct 30 19:15:00.498313 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.498291    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr\" (UID: \"e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr"
Oct 30 19:15:00.530865 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.530417    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q"
Oct 30 19:15:00.555244 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.553806    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnng8\" (UniqueName: \"kubernetes.io/projected/4cbfab36-1855-4785-97e2-4c8a96cd9e0c-kube-api-access-bnng8\") pod \"maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj\" (UID: \"4cbfab36-1855-4785-97e2-4c8a96cd9e0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj"
Oct 30 19:15:00.556440 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.555884    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qps2\" (UniqueName: \"kubernetes.io/projected/4ea62e28-48cf-40e4-9de4-1b767bd839ce-kube-api-access-8qps2\") pod \"maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96\" (UID: \"4ea62e28-48cf-40e4-9de4-1b767bd839ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96"
Oct 30 19:15:00.566328 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.559907    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6c5h\" (UniqueName: \"kubernetes.io/projected/70bd8cf2-4a28-46e2-8223-27e0890f7444-kube-api-access-t6c5h\") pod \"maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx\" (UID: \"70bd8cf2-4a28-46e2-8223-27e0890f7444\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx"
Oct 30 19:15:00.566328 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.560646    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hj8r\" (UniqueName: \"kubernetes.io/projected/534a63d0-c4b9-48cb-88f0-bed24a366017-kube-api-access-5hj8r\") pod \"maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g\" (UID: \"534a63d0-c4b9-48cb-88f0-bed24a366017\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g"
Oct 30 19:15:00.566328 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.561277    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nqv9\" (UniqueName: \"kubernetes.io/projected/7f9d64eb-f439-47cc-8f2c-d7eaf03fd573-kube-api-access-5nqv9\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf\" (UID: \"7f9d64eb-f439-47cc-8f2c-d7eaf03fd573\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf"
Oct 30 19:15:00.566328 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.562632    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6n9h\" (UniqueName: \"kubernetes.io/projected/e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1-kube-api-access-n6n9h\") pod \"maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr\" (UID: \"e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr"
Oct 30 19:15:00.566328 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.563455    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g"
Oct 30 19:15:00.572845 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.572745    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhnl5\" (UniqueName: \"kubernetes.io/projected/d9aa4147-99fd-4ee4-9dd0-43ae21407002-kube-api-access-bhnl5\") pod \"maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh\" (UID: \"d9aa4147-99fd-4ee4-9dd0-43ae21407002\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh"
Oct 30 19:15:00.587659 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.587484    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf"
Oct 30 19:15:00.673470 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.671912    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx"
Oct 30 19:15:00.688940 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.678752    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96"
Oct 30 19:15:00.690564 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.690334    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj"
Oct 30 19:15:00.855901 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.855870    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr"
Oct 30 19:15:00.876351 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:00.876320    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh"
Oct 30 19:15:01.134171 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:01.134012    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4"]
Oct 30 19:15:01.137902 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:15:01.137873    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod000b4c54_e96e_497e_b489_504f2f52094c.slice/crio-a8918783a16b0a888ff7704ffa5029dfcb239974776cd8b4c240fbf00f153fbc WatchSource:0}: Error finding container a8918783a16b0a888ff7704ffa5029dfcb239974776cd8b4c240fbf00f153fbc: Status 404 returned error can't find the container with id a8918783a16b0a888ff7704ffa5029dfcb239974776cd8b4c240fbf00f153fbc
Oct 30 19:15:01.238036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:01.238003    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g"]
Oct 30 19:15:01.366109 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:01.366028    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q"]
Oct 30 19:15:01.371934 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:01.371907    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg"]
Oct 30 19:15:01.375719 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:01.375697    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96"]
Oct 30 19:15:01.430640 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:01.429652    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx"]
Oct 30 19:15:01.433000 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:15:01.432968    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9d33ee4_cc7c_4043_a179_65094f01ca5f.slice/crio-1921d8044b03cc57726adbaba87e3bb8789c204f1ed47b691827a1eb4da721cc WatchSource:0}: Error finding container 1921d8044b03cc57726adbaba87e3bb8789c204f1ed47b691827a1eb4da721cc: Status 404 returned error can't find the container with id 1921d8044b03cc57726adbaba87e3bb8789c204f1ed47b691827a1eb4da721cc
Oct 30 19:15:01.435203 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:15:01.435002    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea62e28_48cf_40e4_9de4_1b767bd839ce.slice/crio-e75a7e90b8abfee55f4943f7a7e33251527e35310eaf24556f3f1b1b89f91818 WatchSource:0}: Error finding container e75a7e90b8abfee55f4943f7a7e33251527e35310eaf24556f3f1b1b89f91818: Status 404 returned error can't find the container with id e75a7e90b8abfee55f4943f7a7e33251527e35310eaf24556f3f1b1b89f91818
Oct 30 19:15:01.473899 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:01.473871    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf"]
Oct 30 19:15:01.479361 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:15:01.479331    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f9d64eb_f439_47cc_8f2c_d7eaf03fd573.slice/crio-8e9888539de7f386f3b8dcf9fc731b306fad510a230fd8f156909c63d7372ff4 WatchSource:0}: Error finding container 8e9888539de7f386f3b8dcf9fc731b306fad510a230fd8f156909c63d7372ff4: Status 404 returned error can't find the container with id 8e9888539de7f386f3b8dcf9fc731b306fad510a230fd8f156909c63d7372ff4
Oct 30 19:15:01.529348 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:01.529296    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr"]
Oct 30 19:15:01.536529 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:15:01.536459    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f86332_e4bb_4eeb_88f6_ab7c4706d6c1.slice/crio-ed90f71ec3bc6cd73ff014b6d4cc4c1b99cfe3a528c01954328dcaa5e8d5351b WatchSource:0}: Error finding container ed90f71ec3bc6cd73ff014b6d4cc4c1b99cfe3a528c01954328dcaa5e8d5351b: Status 404 returned error can't find the container with id ed90f71ec3bc6cd73ff014b6d4cc4c1b99cfe3a528c01954328dcaa5e8d5351b
Oct 30 19:15:01.544342 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:01.540298    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj"]
Oct 30 19:15:01.544342 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:01.544162    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh"]
Oct 30 19:15:01.548007 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:15:01.547608    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cbfab36_1855_4785_97e2_4c8a96cd9e0c.slice/crio-d670423121d7101627ed7c2bf53b4c9138428b2d3eabe90ae52d565416e5f8b2 WatchSource:0}: Error finding container d670423121d7101627ed7c2bf53b4c9138428b2d3eabe90ae52d565416e5f8b2: Status 404 returned error can't find the container with id d670423121d7101627ed7c2bf53b4c9138428b2d3eabe90ae52d565416e5f8b2
Oct 30 19:15:01.908152 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:01.908110    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr" event={"ID":"e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1","Type":"ContainerStarted","Data":"ed90f71ec3bc6cd73ff014b6d4cc4c1b99cfe3a528c01954328dcaa5e8d5351b"}
Oct 30 19:15:01.915345 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:01.915311    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q" event={"ID":"142bd310-9137-4db5-a5e5-27fcf478fbb5","Type":"ContainerStarted","Data":"25996c8034d5d4111f363ba853346ddd2a205b45a87004f165ad7648cf3eefd7"}
Oct 30 19:15:01.918932 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:01.916736    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh" event={"ID":"d9aa4147-99fd-4ee4-9dd0-43ae21407002","Type":"ContainerStarted","Data":"9fce9c60dd9ef9f91048cb94033b83466e490429d0e8863b31e5ad7bc38b752f"}
Oct 30 19:15:01.918932 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:01.918026    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96" event={"ID":"4ea62e28-48cf-40e4-9de4-1b767bd839ce","Type":"ContainerStarted","Data":"e75a7e90b8abfee55f4943f7a7e33251527e35310eaf24556f3f1b1b89f91818"}
Oct 30 19:15:01.925141 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:01.921484    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg" event={"ID":"d9d33ee4-cc7c-4043-a179-65094f01ca5f","Type":"ContainerStarted","Data":"1921d8044b03cc57726adbaba87e3bb8789c204f1ed47b691827a1eb4da721cc"}
Oct 30 19:15:01.934977 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:01.934948    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx" event={"ID":"70bd8cf2-4a28-46e2-8223-27e0890f7444","Type":"ContainerStarted","Data":"687a88ebc5ccf839cb6f9ec91ceb5a1d248e463023ef8ac6f2822c2c41a94ce0"}
Oct 30 19:15:01.938420 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:01.938372    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf" event={"ID":"7f9d64eb-f439-47cc-8f2c-d7eaf03fd573","Type":"ContainerStarted","Data":"8e9888539de7f386f3b8dcf9fc731b306fad510a230fd8f156909c63d7372ff4"}
Oct 30 19:15:01.940102 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:01.940048    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj" event={"ID":"4cbfab36-1855-4785-97e2-4c8a96cd9e0c","Type":"ContainerStarted","Data":"d670423121d7101627ed7c2bf53b4c9138428b2d3eabe90ae52d565416e5f8b2"}
Oct 30 19:15:01.941849 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:01.941820    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4" event={"ID":"000b4c54-e96e-497e-b489-504f2f52094c","Type":"ContainerStarted","Data":"4cafe108d0b43b47dd0b4b2c8ba48dfd35d518438c755c51b0f5b0fa3801e5a8"}
Oct 30 19:15:01.941936 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:01.941857    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4" event={"ID":"000b4c54-e96e-497e-b489-504f2f52094c","Type":"ContainerStarted","Data":"a8918783a16b0a888ff7704ffa5029dfcb239974776cd8b4c240fbf00f153fbc"}
Oct 30 19:15:01.944020 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:01.944001    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g" event={"ID":"534a63d0-c4b9-48cb-88f0-bed24a366017","Type":"ContainerStarted","Data":"b19adfa4e27d627edc85b5fd3a303700d36b734d80655f6941f632796fd0c99d"}
Oct 30 19:15:01.944116 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:01.944035    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g" event={"ID":"534a63d0-c4b9-48cb-88f0-bed24a366017","Type":"ContainerStarted","Data":"1aac7f0d0252396cb0f37011ca326a2cd7a7edbe59c76ea8cff2a61653b8416e"}
Oct 30 19:15:02.016338 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:02.016295    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:02.049709 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:02.049679    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4"]
Oct 30 19:15:02.065129 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:02.064983    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4" podStartSLOduration=2.064965924 podStartE2EDuration="2.064965924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.060083116 +0000 UTC m=+2303.337133558" watchObservedRunningTime="2026-10-30 19:15:02.064965924 +0000 UTC m=+2303.342016377"
Oct 30 19:15:02.171368 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:02.171309    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g" podStartSLOduration=2.171296436 podStartE2EDuration="2.171296436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:02.170223104 +0000 UTC m=+2303.447273549" watchObservedRunningTime="2026-10-30 19:15:02.171296436 +0000 UTC m=+2303.448346884"
Oct 30 19:15:03.009694 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:03.009644    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj" event={"ID":"4cbfab36-1855-4785-97e2-4c8a96cd9e0c","Type":"ContainerStarted","Data":"3d74b2b600a7b23e81e543a601a3d62d1b59ddf810652428251dc77b98ce7597"}
Oct 30 19:15:03.012816 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:03.012787    4924 generic.go:356] "Generic (PLEG): container finished" podID="000b4c54-e96e-497e-b489-504f2f52094c" containerID="4cafe108d0b43b47dd0b4b2c8ba48dfd35d518438c755c51b0f5b0fa3801e5a8" exitCode=6
Oct 30 19:15:03.013011 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:03.012973    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4" event={"ID":"000b4c54-e96e-497e-b489-504f2f52094c","Type":"ContainerDied","Data":"4cafe108d0b43b47dd0b4b2c8ba48dfd35d518438c755c51b0f5b0fa3801e5a8"}
Oct 30 19:15:03.013582 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:03.013439    4924 scope.go:122] "RemoveContainer" containerID="4cafe108d0b43b47dd0b4b2c8ba48dfd35d518438c755c51b0f5b0fa3801e5a8"
Oct 30 19:15:03.017163 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:03.016341    4924 generic.go:356] "Generic (PLEG): container finished" podID="534a63d0-c4b9-48cb-88f0-bed24a366017" containerID="b19adfa4e27d627edc85b5fd3a303700d36b734d80655f6941f632796fd0c99d" exitCode=60
Oct 30 19:15:03.017163 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:03.016476    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g" event={"ID":"534a63d0-c4b9-48cb-88f0-bed24a366017","Type":"ContainerDied","Data":"b19adfa4e27d627edc85b5fd3a303700d36b734d80655f6941f632796fd0c99d"}
Oct 30 19:15:03.017163 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:03.016907    4924 scope.go:122] "RemoveContainer" containerID="b19adfa4e27d627edc85b5fd3a303700d36b734d80655f6941f632796fd0c99d"
Oct 30 19:15:03.058901 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:03.058877    4924 scope.go:122] "RemoveContainer" containerID="4f786ac70721f0e5a9b1ae3251aac87c0fe1134a567db55b14fc763c6372c0c5"
Oct 30 19:15:03.059477 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:03.059444    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr" event={"ID":"e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1","Type":"ContainerStarted","Data":"4f786ac70721f0e5a9b1ae3251aac87c0fe1134a567db55b14fc763c6372c0c5"}
Oct 30 19:15:03.063052 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:03.062973    4924 generic.go:356] "Generic (PLEG): container finished" podID="142bd310-9137-4db5-a5e5-27fcf478fbb5" containerID="fade34d7a81d5e7946bdafea885fd6479a83b22ab6d0c4590a64769baaa70d48" exitCode=60
Oct 30 19:15:03.063241 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:03.063215    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q" event={"ID":"142bd310-9137-4db5-a5e5-27fcf478fbb5","Type":"ContainerDied","Data":"fade34d7a81d5e7946bdafea885fd6479a83b22ab6d0c4590a64769baaa70d48"}
Oct 30 19:15:03.063529 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:03.063509    4924 scope.go:122] "RemoveContainer" containerID="fade34d7a81d5e7946bdafea885fd6479a83b22ab6d0c4590a64769baaa70d48"
Oct 30 19:15:03.065963 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:03.065928    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh" event={"ID":"d9aa4147-99fd-4ee4-9dd0-43ae21407002","Type":"ContainerStarted","Data":"922704a79299b23b97421cecfba76feeddc927b7ca7ad3c9f0f8c3733e21e9ad"}
Oct 30 19:15:03.066320 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:03.066303    4924 scope.go:122] "RemoveContainer" containerID="922704a79299b23b97421cecfba76feeddc927b7ca7ad3c9f0f8c3733e21e9ad"
Oct 30 19:15:03.068241 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:03.068219    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96" event={"ID":"4ea62e28-48cf-40e4-9de4-1b767bd839ce","Type":"ContainerStarted","Data":"2419d4805189de2f53f927df8f710f71ce54f49010ba24b0f7518f308ed87958"}
Oct 30 19:15:03.068476 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:03.068460    4924 scope.go:122] "RemoveContainer" containerID="2419d4805189de2f53f927df8f710f71ce54f49010ba24b0f7518f308ed87958"
Oct 30 19:15:03.069960 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:03.069935    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg" event={"ID":"d9d33ee4-cc7c-4043-a179-65094f01ca5f","Type":"ContainerStarted","Data":"8addb0777c1172e27767387ed425dc545a12639fef26965e4ee08094651d7f88"}
Oct 30 19:15:03.070482 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:03.070135    4924 scope.go:122] "RemoveContainer" containerID="8addb0777c1172e27767387ed425dc545a12639fef26965e4ee08094651d7f88"
Oct 30 19:15:03.072665 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:03.072639    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx" event={"ID":"70bd8cf2-4a28-46e2-8223-27e0890f7444","Type":"ContainerStarted","Data":"7891fdd18fce70c889311cf6a82483b9bc761ed64e21982273ee4560cbb0caae"}
Oct 30 19:15:03.072858 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:03.072839    4924 scope.go:122] "RemoveContainer" containerID="7891fdd18fce70c889311cf6a82483b9bc761ed64e21982273ee4560cbb0caae"
Oct 30 19:15:03.074362 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:03.074314    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf" event={"ID":"7f9d64eb-f439-47cc-8f2c-d7eaf03fd573","Type":"ContainerStarted","Data":"acec896ec2f39aeb2f118de61eca3a48cd9e15989481ef612a8996a3a044e0c3"}
Oct 30 19:15:03.074584 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:03.074567    4924 scope.go:122] "RemoveContainer" containerID="acec896ec2f39aeb2f118de61eca3a48cd9e15989481ef612a8996a3a044e0c3"
Oct 30 19:15:03.121606 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:03.121558    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj" podStartSLOduration=3.121541357 podStartE2EDuration="3.121541357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:03.119838703 +0000 UTC m=+2304.396889147" watchObservedRunningTime="2026-10-30 19:15:03.121541357 +0000 UTC m=+2304.398591826"
Oct 30 19:15:03.377498 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:03.377461    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc" podUID="0b676034-384c-4c43-a8f5-991977f109ce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:15:03.706789 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:03.706731    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:04.085821 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.085782    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4" event={"ID":"000b4c54-e96e-497e-b489-504f2f52094c","Type":"ContainerStarted","Data":"3094c364ff22db71cf05fe33694e8b1c1b5db75e9ce72733b74edf35d8a89a88"}
Oct 30 19:15:04.086298 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.085870    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4" podUID="000b4c54-e96e-497e-b489-504f2f52094c" containerName="cleanup" containerID="cri-o://3094c364ff22db71cf05fe33694e8b1c1b5db75e9ce72733b74edf35d8a89a88" gracePeriod=30
Oct 30 19:15:04.088117 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.088052    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g" event={"ID":"534a63d0-c4b9-48cb-88f0-bed24a366017","Type":"ContainerStarted","Data":"689804f1c62420b6997dd80a71df8531bf7e6d9c8804f87f7f689ad0d93834b7"}
Oct 30 19:15:04.090256 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.090234    4924 generic.go:356] "Generic (PLEG): container finished" podID="e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1" containerID="4f786ac70721f0e5a9b1ae3251aac87c0fe1134a567db55b14fc763c6372c0c5" exitCode=60
Oct 30 19:15:04.090394 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.090315    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr" event={"ID":"e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1","Type":"ContainerDied","Data":"4f786ac70721f0e5a9b1ae3251aac87c0fe1134a567db55b14fc763c6372c0c5"}
Oct 30 19:15:04.090394 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.090366    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr" event={"ID":"e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1","Type":"ContainerStarted","Data":"e880e44d5bd8cf65996036201caa9aff1e994be589d9fcb42526397c3c9d72a5"}
Oct 30 19:15:04.092428 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.092402    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q" event={"ID":"142bd310-9137-4db5-a5e5-27fcf478fbb5","Type":"ContainerStarted","Data":"8abffc49dc96cbf26716d2ae6d38998cdd29acb4dd1fd8ad423403b31b579b6d"}
Oct 30 19:15:04.094437 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.094412    4924 generic.go:356] "Generic (PLEG): container finished" podID="d9aa4147-99fd-4ee4-9dd0-43ae21407002" containerID="922704a79299b23b97421cecfba76feeddc927b7ca7ad3c9f0f8c3733e21e9ad" exitCode=60
Oct 30 19:15:04.094555 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.094507    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh" event={"ID":"d9aa4147-99fd-4ee4-9dd0-43ae21407002","Type":"ContainerDied","Data":"922704a79299b23b97421cecfba76feeddc927b7ca7ad3c9f0f8c3733e21e9ad"}
Oct 30 19:15:04.094626 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.094555    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh" event={"ID":"d9aa4147-99fd-4ee4-9dd0-43ae21407002","Type":"ContainerStarted","Data":"e20c8655236127359e725e37333b40bce3b4302869e58d26f822ce947f0698d7"}
Oct 30 19:15:04.096375 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.096354    4924 generic.go:356] "Generic (PLEG): container finished" podID="4ea62e28-48cf-40e4-9de4-1b767bd839ce" containerID="2419d4805189de2f53f927df8f710f71ce54f49010ba24b0f7518f308ed87958" exitCode=60
Oct 30 19:15:04.096482 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.096432    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96" event={"ID":"4ea62e28-48cf-40e4-9de4-1b767bd839ce","Type":"ContainerDied","Data":"2419d4805189de2f53f927df8f710f71ce54f49010ba24b0f7518f308ed87958"}
Oct 30 19:15:04.096545 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.096479    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96" event={"ID":"4ea62e28-48cf-40e4-9de4-1b767bd839ce","Type":"ContainerStarted","Data":"b9c2353176b0df07e91c4cb363b0ec4b3727b60254b491690f5277dc4523158a"}
Oct 30 19:15:04.097627 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.097608    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:04.098487 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.098467    4924 generic.go:356] "Generic (PLEG): container finished" podID="d9d33ee4-cc7c-4043-a179-65094f01ca5f" containerID="8addb0777c1172e27767387ed425dc545a12639fef26965e4ee08094651d7f88" exitCode=60
Oct 30 19:15:04.098605 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.098536    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg" event={"ID":"d9d33ee4-cc7c-4043-a179-65094f01ca5f","Type":"ContainerDied","Data":"8addb0777c1172e27767387ed425dc545a12639fef26965e4ee08094651d7f88"}
Oct 30 19:15:04.098605 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.098582    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg" event={"ID":"d9d33ee4-cc7c-4043-a179-65094f01ca5f","Type":"ContainerStarted","Data":"9cef2a28ad870123b7e723f8b149aa8952716ebf53cdb97c1e5059030e447f05"}
Oct 30 19:15:04.100493 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.100473    4924 generic.go:356] "Generic (PLEG): container finished" podID="70bd8cf2-4a28-46e2-8223-27e0890f7444" containerID="7891fdd18fce70c889311cf6a82483b9bc761ed64e21982273ee4560cbb0caae" exitCode=60
Oct 30 19:15:04.100600 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.100578    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx" event={"ID":"70bd8cf2-4a28-46e2-8223-27e0890f7444","Type":"ContainerDied","Data":"7891fdd18fce70c889311cf6a82483b9bc761ed64e21982273ee4560cbb0caae"}
Oct 30 19:15:04.100710 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.100622    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx" event={"ID":"70bd8cf2-4a28-46e2-8223-27e0890f7444","Type":"ContainerStarted","Data":"f6992130696f4be8093f28f8409e9eac848b44c35738d787c01c332a251c9c70"}
Oct 30 19:15:04.104270 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.104249    4924 generic.go:356] "Generic (PLEG): container finished" podID="7f9d64eb-f439-47cc-8f2c-d7eaf03fd573" containerID="acec896ec2f39aeb2f118de61eca3a48cd9e15989481ef612a8996a3a044e0c3" exitCode=60
Oct 30 19:15:04.104499 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.104319    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf" event={"ID":"7f9d64eb-f439-47cc-8f2c-d7eaf03fd573","Type":"ContainerDied","Data":"acec896ec2f39aeb2f118de61eca3a48cd9e15989481ef612a8996a3a044e0c3"}
Oct 30 19:15:04.104586 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.104529    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf" event={"ID":"7f9d64eb-f439-47cc-8f2c-d7eaf03fd573","Type":"ContainerStarted","Data":"eb717f40124cc0f021391ed892d2f183131db2b021b55423dfb41d591904158c"}
Oct 30 19:15:04.106584 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.106558    4924 generic.go:356] "Generic (PLEG): container finished" podID="4cbfab36-1855-4785-97e2-4c8a96cd9e0c" containerID="3d74b2b600a7b23e81e543a601a3d62d1b59ddf810652428251dc77b98ce7597" exitCode=60
Oct 30 19:15:04.106678 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.106593    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj" event={"ID":"4cbfab36-1855-4785-97e2-4c8a96cd9e0c","Type":"ContainerDied","Data":"3d74b2b600a7b23e81e543a601a3d62d1b59ddf810652428251dc77b98ce7597"}
Oct 30 19:15:04.106848 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.106834    4924 scope.go:122] "RemoveContainer" containerID="3d74b2b600a7b23e81e543a601a3d62d1b59ddf810652428251dc77b98ce7597"
Oct 30 19:15:04.185762 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.185645    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc"]
Oct 30 19:15:04.191798 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.191765    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-091e8f-txvvc"]
Oct 30 19:15:04.223935 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.223877    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96" podStartSLOduration=4.223859149 podStartE2EDuration="4.223859149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:04.223055424 +0000 UTC m=+2305.500105869" watchObservedRunningTime="2026-10-30 19:15:04.223859149 +0000 UTC m=+2305.500909594"
Oct 30 19:15:04.291783 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.291749    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:04.440287 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.440210    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx" podStartSLOduration=4.440196254 podStartE2EDuration="4.440196254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:04.436889159 +0000 UTC m=+2305.713939604" watchObservedRunningTime="2026-10-30 19:15:04.440196254 +0000 UTC m=+2305.717246698"
Oct 30 19:15:04.544754 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.544300    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf" podStartSLOduration=4.544282247 podStartE2EDuration="4.544282247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:04.544193844 +0000 UTC m=+2305.821244291" watchObservedRunningTime="2026-10-30 19:15:04.544282247 +0000 UTC m=+2305.821332693"
Oct 30 19:15:04.652453 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.652396    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q" podStartSLOduration=4.652381174 podStartE2EDuration="4.652381174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:04.651392633 +0000 UTC m=+2305.928443073" watchObservedRunningTime="2026-10-30 19:15:04.652381174 +0000 UTC m=+2305.929431619"
Oct 30 19:15:04.661768 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.661704    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:04.837190 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.837159    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62be35dd-f037-4b33-b11a-d1aaf486af36" path="/var/lib/kubelet/pods/62be35dd-f037-4b33-b11a-d1aaf486af36/volumes"
Oct 30 19:15:04.872879 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.872826    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr" podStartSLOduration=4.872804285 podStartE2EDuration="4.872804285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:04.868308052 +0000 UTC m=+2306.145358497" watchObservedRunningTime="2026-10-30 19:15:04.872804285 +0000 UTC m=+2306.149854732"
Oct 30 19:15:04.996623 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:04.996552    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4"
Oct 30 19:15:05.008470 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.008444    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/000b4c54-e96e-497e-b489-504f2f52094c-kube-api-access-kdw2f\" (UniqueName: \"kubernetes.io/projected/000b4c54-e96e-497e-b489-504f2f52094c-kube-api-access-kdw2f\") pod \"000b4c54-e96e-497e-b489-504f2f52094c\" (UID: \"000b4c54-e96e-497e-b489-504f2f52094c\") "
Oct 30 19:15:05.008808 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.008719    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/000b4c54-e96e-497e-b489-504f2f52094c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/000b4c54-e96e-497e-b489-504f2f52094c-maas-api-service-ca\") pod \"000b4c54-e96e-497e-b489-504f2f52094c\" (UID: \"000b4c54-e96e-497e-b489-504f2f52094c\") "
Oct 30 19:15:05.009049 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.009018    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/000b4c54-e96e-497e-b489-504f2f52094c-maas-api-service-ca" pod "000b4c54-e96e-497e-b489-504f2f52094c" (UID: "000b4c54-e96e-497e-b489-504f2f52094c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:05.009271 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.009253    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/000b4c54-e96e-497e-b489-504f2f52094c-maas-api-service-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:05.010561 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.010537    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/000b4c54-e96e-497e-b489-504f2f52094c-kube-api-access-kdw2f" pod "000b4c54-e96e-497e-b489-504f2f52094c" (UID: "000b4c54-e96e-497e-b489-504f2f52094c"). InnerVolumeSpecName "kube-api-access-kdw2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:05.086484 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.086430    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh" podStartSLOduration=5.086416334 podStartE2EDuration="5.086416334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:05.082620307 +0000 UTC m=+2306.359670753" watchObservedRunningTime="2026-10-30 19:15:05.086416334 +0000 UTC m=+2306.363466780"
Oct 30 19:15:05.111035 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.111007    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdw2f\" (UniqueName: \"kubernetes.io/projected/000b4c54-e96e-497e-b489-504f2f52094c-kube-api-access-kdw2f\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:05.119272 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.119243    4924 generic.go:356] "Generic (PLEG): container finished" podID="000b4c54-e96e-497e-b489-504f2f52094c" containerID="3094c364ff22db71cf05fe33694e8b1c1b5db75e9ce72733b74edf35d8a89a88" exitCode=6
Oct 30 19:15:05.119367 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.119299    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4"
Oct 30 19:15:05.119367 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.119318    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4" event={"ID":"000b4c54-e96e-497e-b489-504f2f52094c","Type":"ContainerDied","Data":"3094c364ff22db71cf05fe33694e8b1c1b5db75e9ce72733b74edf35d8a89a88"}
Oct 30 19:15:05.119482 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.119380    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4" event={"ID":"000b4c54-e96e-497e-b489-504f2f52094c","Type":"ContainerDied","Data":"a8918783a16b0a888ff7704ffa5029dfcb239974776cd8b4c240fbf00f153fbc"}
Oct 30 19:15:05.119482 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.119404    4924 scope.go:122] "RemoveContainer" containerID="3094c364ff22db71cf05fe33694e8b1c1b5db75e9ce72733b74edf35d8a89a88"
Oct 30 19:15:05.121653 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.121633    4924 generic.go:356] "Generic (PLEG): container finished" podID="534a63d0-c4b9-48cb-88f0-bed24a366017" containerID="689804f1c62420b6997dd80a71df8531bf7e6d9c8804f87f7f689ad0d93834b7" exitCode=60
Oct 30 19:15:05.121993 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.121762    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g" event={"ID":"534a63d0-c4b9-48cb-88f0-bed24a366017","Type":"ContainerDied","Data":"689804f1c62420b6997dd80a71df8531bf7e6d9c8804f87f7f689ad0d93834b7"}
Oct 30 19:15:05.122098 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.121993    4924 scope.go:122] "RemoveContainer" containerID="689804f1c62420b6997dd80a71df8531bf7e6d9c8804f87f7f689ad0d93834b7"
Oct 30 19:15:05.122357 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:05.122334    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g_odh-ai-gateway-infra(534a63d0-c4b9-48cb-88f0-bed24a366017)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g" podUID="534a63d0-c4b9-48cb-88f0-bed24a366017"
Oct 30 19:15:05.123792 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.123772    4924 generic.go:356] "Generic (PLEG): container finished" podID="e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1" containerID="e880e44d5bd8cf65996036201caa9aff1e994be589d9fcb42526397c3c9d72a5" exitCode=60
Oct 30 19:15:05.123897 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.123853    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr" event={"ID":"e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1","Type":"ContainerDied","Data":"e880e44d5bd8cf65996036201caa9aff1e994be589d9fcb42526397c3c9d72a5"}
Oct 30 19:15:05.124121 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.124104    4924 scope.go:122] "RemoveContainer" containerID="e880e44d5bd8cf65996036201caa9aff1e994be589d9fcb42526397c3c9d72a5"
Oct 30 19:15:05.124409 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:05.124384    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr_odh-ai-gateway-infra(e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr" podUID="e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1"
Oct 30 19:15:05.125885 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.125861    4924 generic.go:356] "Generic (PLEG): container finished" podID="142bd310-9137-4db5-a5e5-27fcf478fbb5" containerID="8abffc49dc96cbf26716d2ae6d38998cdd29acb4dd1fd8ad423403b31b579b6d" exitCode=60
Oct 30 19:15:05.125975 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.125931    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q" event={"ID":"142bd310-9137-4db5-a5e5-27fcf478fbb5","Type":"ContainerDied","Data":"8abffc49dc96cbf26716d2ae6d38998cdd29acb4dd1fd8ad423403b31b579b6d"}
Oct 30 19:15:05.126163 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.126147    4924 scope.go:122] "RemoveContainer" containerID="8abffc49dc96cbf26716d2ae6d38998cdd29acb4dd1fd8ad423403b31b579b6d"
Oct 30 19:15:05.126394 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:05.126367    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889795-bjw2q_odh-ai-gateway-infra(142bd310-9137-4db5-a5e5-27fcf478fbb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q" podUID="142bd310-9137-4db5-a5e5-27fcf478fbb5"
Oct 30 19:15:05.127901 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.127883    4924 generic.go:356] "Generic (PLEG): container finished" podID="d9aa4147-99fd-4ee4-9dd0-43ae21407002" containerID="e20c8655236127359e725e37333b40bce3b4302869e58d26f822ce947f0698d7" exitCode=60
Oct 30 19:15:05.127990 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.127906    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh" event={"ID":"d9aa4147-99fd-4ee4-9dd0-43ae21407002","Type":"ContainerDied","Data":"e20c8655236127359e725e37333b40bce3b4302869e58d26f822ce947f0698d7"}
Oct 30 19:15:05.128230 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.128211    4924 scope.go:122] "RemoveContainer" containerID="e20c8655236127359e725e37333b40bce3b4302869e58d26f822ce947f0698d7"
Oct 30 19:15:05.128453 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:05.128433    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh_odh-ai-gateway-infra(d9aa4147-99fd-4ee4-9dd0-43ae21407002)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh" podUID="d9aa4147-99fd-4ee4-9dd0-43ae21407002"
Oct 30 19:15:05.129783 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.129764    4924 generic.go:356] "Generic (PLEG): container finished" podID="4ea62e28-48cf-40e4-9de4-1b767bd839ce" containerID="b9c2353176b0df07e91c4cb363b0ec4b3727b60254b491690f5277dc4523158a" exitCode=60
Oct 30 19:15:05.129860 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.129821    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96" event={"ID":"4ea62e28-48cf-40e4-9de4-1b767bd839ce","Type":"ContainerDied","Data":"b9c2353176b0df07e91c4cb363b0ec4b3727b60254b491690f5277dc4523158a"}
Oct 30 19:15:05.130175 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.130155    4924 scope.go:122] "RemoveContainer" containerID="b9c2353176b0df07e91c4cb363b0ec4b3727b60254b491690f5277dc4523158a"
Oct 30 19:15:05.130440 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:05.130418    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96_odh-ai-gateway-infra(4ea62e28-48cf-40e4-9de4-1b767bd839ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96" podUID="4ea62e28-48cf-40e4-9de4-1b767bd839ce"
Oct 30 19:15:05.131584 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.131564    4924 generic.go:356] "Generic (PLEG): container finished" podID="d9d33ee4-cc7c-4043-a179-65094f01ca5f" containerID="9cef2a28ad870123b7e723f8b149aa8952716ebf53cdb97c1e5059030e447f05" exitCode=60
Oct 30 19:15:05.131675 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.131645    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg" event={"ID":"d9d33ee4-cc7c-4043-a179-65094f01ca5f","Type":"ContainerDied","Data":"9cef2a28ad870123b7e723f8b149aa8952716ebf53cdb97c1e5059030e447f05"}
Oct 30 19:15:05.131816 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.131801    4924 scope.go:122] "RemoveContainer" containerID="9cef2a28ad870123b7e723f8b149aa8952716ebf53cdb97c1e5059030e447f05"
Oct 30 19:15:05.132025 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:05.131989    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg_odh-ai-gateway-infra(d9d33ee4-cc7c-4043-a179-65094f01ca5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg" podUID="d9d33ee4-cc7c-4043-a179-65094f01ca5f"
Oct 30 19:15:05.133469 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.133443    4924 generic.go:356] "Generic (PLEG): container finished" podID="70bd8cf2-4a28-46e2-8223-27e0890f7444" containerID="f6992130696f4be8093f28f8409e9eac848b44c35738d787c01c332a251c9c70" exitCode=60
Oct 30 19:15:05.133551 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.133524    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx" event={"ID":"70bd8cf2-4a28-46e2-8223-27e0890f7444","Type":"ContainerDied","Data":"f6992130696f4be8093f28f8409e9eac848b44c35738d787c01c332a251c9c70"}
Oct 30 19:15:05.133857 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.133836    4924 scope.go:122] "RemoveContainer" containerID="f6992130696f4be8093f28f8409e9eac848b44c35738d787c01c332a251c9c70"
Oct 30 19:15:05.134149 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:05.134124    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx_odh-ai-gateway-infra(70bd8cf2-4a28-46e2-8223-27e0890f7444)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx" podUID="70bd8cf2-4a28-46e2-8223-27e0890f7444"
Oct 30 19:15:05.135446 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.135427    4924 generic.go:356] "Generic (PLEG): container finished" podID="7f9d64eb-f439-47cc-8f2c-d7eaf03fd573" containerID="eb717f40124cc0f021391ed892d2f183131db2b021b55423dfb41d591904158c" exitCode=60
Oct 30 19:15:05.135541 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.135506    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf" event={"ID":"7f9d64eb-f439-47cc-8f2c-d7eaf03fd573","Type":"ContainerDied","Data":"eb717f40124cc0f021391ed892d2f183131db2b021b55423dfb41d591904158c"}
Oct 30 19:15:05.135693 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.135675    4924 scope.go:122] "RemoveContainer" containerID="eb717f40124cc0f021391ed892d2f183131db2b021b55423dfb41d591904158c"
Oct 30 19:15:05.135929 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:05.135909    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf_odh-ai-gateway-infra(7f9d64eb-f439-47cc-8f2c-d7eaf03fd573)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf" podUID="7f9d64eb-f439-47cc-8f2c-d7eaf03fd573"
Oct 30 19:15:05.137477 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.137447    4924 generic.go:356] "Generic (PLEG): container finished" podID="4cbfab36-1855-4785-97e2-4c8a96cd9e0c" containerID="b5c303d5aff1f7944d240f77f7b74ecc19bf220b56a2fd847fccb9905852389b" exitCode=60
Oct 30 19:15:05.137580 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.137491    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj" event={"ID":"4cbfab36-1855-4785-97e2-4c8a96cd9e0c","Type":"ContainerDied","Data":"b5c303d5aff1f7944d240f77f7b74ecc19bf220b56a2fd847fccb9905852389b"}
Oct 30 19:15:05.137733 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.137717    4924 scope.go:122] "RemoveContainer" containerID="b5c303d5aff1f7944d240f77f7b74ecc19bf220b56a2fd847fccb9905852389b"
Oct 30 19:15:05.137938 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:05.137917    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj_odh-ai-gateway-infra(4cbfab36-1855-4785-97e2-4c8a96cd9e0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj" podUID="4cbfab36-1855-4785-97e2-4c8a96cd9e0c"
Oct 30 19:15:05.161803 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.161780    4924 scope.go:122] "RemoveContainer" containerID="4cafe108d0b43b47dd0b4b2c8ba48dfd35d518438c755c51b0f5b0fa3801e5a8"
Oct 30 19:15:05.172701 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.172675    4924 scope.go:122] "RemoveContainer" containerID="3094c364ff22db71cf05fe33694e8b1c1b5db75e9ce72733b74edf35d8a89a88"
Oct 30 19:15:05.173008 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:05.172988    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3094c364ff22db71cf05fe33694e8b1c1b5db75e9ce72733b74edf35d8a89a88\": container with ID starting with 3094c364ff22db71cf05fe33694e8b1c1b5db75e9ce72733b74edf35d8a89a88 not found: ID does not exist" containerID="3094c364ff22db71cf05fe33694e8b1c1b5db75e9ce72733b74edf35d8a89a88"
Oct 30 19:15:05.173193 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.173015    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3094c364ff22db71cf05fe33694e8b1c1b5db75e9ce72733b74edf35d8a89a88"} err="failed to get container status \"3094c364ff22db71cf05fe33694e8b1c1b5db75e9ce72733b74edf35d8a89a88\": rpc error: code = NotFound desc = could not find container \"3094c364ff22db71cf05fe33694e8b1c1b5db75e9ce72733b74edf35d8a89a88\": container with ID starting with 3094c364ff22db71cf05fe33694e8b1c1b5db75e9ce72733b74edf35d8a89a88 not found: ID does not exist"
Oct 30 19:15:05.173193 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.173033    4924 scope.go:122] "RemoveContainer" containerID="4cafe108d0b43b47dd0b4b2c8ba48dfd35d518438c755c51b0f5b0fa3801e5a8"
Oct 30 19:15:05.173304 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:05.173278    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cafe108d0b43b47dd0b4b2c8ba48dfd35d518438c755c51b0f5b0fa3801e5a8\": container with ID starting with 4cafe108d0b43b47dd0b4b2c8ba48dfd35d518438c755c51b0f5b0fa3801e5a8 not found: ID does not exist" containerID="4cafe108d0b43b47dd0b4b2c8ba48dfd35d518438c755c51b0f5b0fa3801e5a8"
Oct 30 19:15:05.173339 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.173303    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cafe108d0b43b47dd0b4b2c8ba48dfd35d518438c755c51b0f5b0fa3801e5a8"} err="failed to get container status \"4cafe108d0b43b47dd0b4b2c8ba48dfd35d518438c755c51b0f5b0fa3801e5a8\": rpc error: code = NotFound desc = could not find container \"4cafe108d0b43b47dd0b4b2c8ba48dfd35d518438c755c51b0f5b0fa3801e5a8\": container with ID starting with 4cafe108d0b43b47dd0b4b2c8ba48dfd35d518438c755c51b0f5b0fa3801e5a8 not found: ID does not exist"
Oct 30 19:15:05.173339 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.173318    4924 scope.go:122] "RemoveContainer" containerID="b19adfa4e27d627edc85b5fd3a303700d36b734d80655f6941f632796fd0c99d"
Oct 30 19:15:05.188834 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.188813    4924 scope.go:122] "RemoveContainer" containerID="4f786ac70721f0e5a9b1ae3251aac87c0fe1134a567db55b14fc763c6372c0c5"
Oct 30 19:15:05.202923 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.202901    4924 scope.go:122] "RemoveContainer" containerID="fade34d7a81d5e7946bdafea885fd6479a83b22ab6d0c4590a64769baaa70d48"
Oct 30 19:15:05.203116 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.203054    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg" podStartSLOduration=5.203041094 podStartE2EDuration="5.203041094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:05.199155057 +0000 UTC m=+2306.476205502" watchObservedRunningTime="2026-10-30 19:15:05.203041094 +0000 UTC m=+2306.480091542"
Oct 30 19:15:05.218987 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.218959    4924 scope.go:122] "RemoveContainer" containerID="922704a79299b23b97421cecfba76feeddc927b7ca7ad3c9f0f8c3733e21e9ad"
Oct 30 19:15:05.234684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.234662    4924 scope.go:122] "RemoveContainer" containerID="2419d4805189de2f53f927df8f710f71ce54f49010ba24b0f7518f308ed87958"
Oct 30 19:15:05.250144 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.250122    4924 scope.go:122] "RemoveContainer" containerID="8addb0777c1172e27767387ed425dc545a12639fef26965e4ee08094651d7f88"
Oct 30 19:15:05.263835 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.263812    4924 scope.go:122] "RemoveContainer" containerID="7891fdd18fce70c889311cf6a82483b9bc761ed64e21982273ee4560cbb0caae"
Oct 30 19:15:05.280042 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.280023    4924 scope.go:122] "RemoveContainer" containerID="acec896ec2f39aeb2f118de61eca3a48cd9e15989481ef612a8996a3a044e0c3"
Oct 30 19:15:05.295864 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.295840    4924 scope.go:122] "RemoveContainer" containerID="3d74b2b600a7b23e81e543a601a3d62d1b59ddf810652428251dc77b98ce7597"
Oct 30 19:15:05.559765 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.559730    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl"]
Oct 30 19:15:05.579461 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.579418    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4"]
Oct 30 19:15:05.583576 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.583544    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-091e8f-29889795-d86w4"]
Oct 30 19:15:05.670319 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.669354    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb"]
Oct 30 19:15:05.980365 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:05.965760    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:06.147859 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:06.147829    4924 scope.go:122] "RemoveContainer" containerID="f6992130696f4be8093f28f8409e9eac848b44c35738d787c01c332a251c9c70"
Oct 30 19:15:06.148289 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:06.148216    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx_odh-ai-gateway-infra(70bd8cf2-4a28-46e2-8223-27e0890f7444)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx" podUID="70bd8cf2-4a28-46e2-8223-27e0890f7444"
Oct 30 19:15:06.149355 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:06.149334    4924 scope.go:122] "RemoveContainer" containerID="eb717f40124cc0f021391ed892d2f183131db2b021b55423dfb41d591904158c"
Oct 30 19:15:06.149686 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:06.149664    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf_odh-ai-gateway-infra(7f9d64eb-f439-47cc-8f2c-d7eaf03fd573)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf" podUID="7f9d64eb-f439-47cc-8f2c-d7eaf03fd573"
Oct 30 19:15:06.150891 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:06.150864    4924 scope.go:122] "RemoveContainer" containerID="b5c303d5aff1f7944d240f77f7b74ecc19bf220b56a2fd847fccb9905852389b"
Oct 30 19:15:06.151262 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:06.151237    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj_odh-ai-gateway-infra(4cbfab36-1855-4785-97e2-4c8a96cd9e0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj" podUID="4cbfab36-1855-4785-97e2-4c8a96cd9e0c"
Oct 30 19:15:06.153252 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:06.153234    4924 scope.go:122] "RemoveContainer" containerID="689804f1c62420b6997dd80a71df8531bf7e6d9c8804f87f7f689ad0d93834b7"
Oct 30 19:15:06.153486 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:06.153464    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g_odh-ai-gateway-infra(534a63d0-c4b9-48cb-88f0-bed24a366017)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g" podUID="534a63d0-c4b9-48cb-88f0-bed24a366017"
Oct 30 19:15:06.154690 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:06.154674    4924 scope.go:122] "RemoveContainer" containerID="e880e44d5bd8cf65996036201caa9aff1e994be589d9fcb42526397c3c9d72a5"
Oct 30 19:15:06.154936 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:06.154913    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr_odh-ai-gateway-infra(e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr" podUID="e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1"
Oct 30 19:15:06.156124 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:06.156107    4924 scope.go:122] "RemoveContainer" containerID="8abffc49dc96cbf26716d2ae6d38998cdd29acb4dd1fd8ad423403b31b579b6d"
Oct 30 19:15:06.156337 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:06.156303    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889795-bjw2q_odh-ai-gateway-infra(142bd310-9137-4db5-a5e5-27fcf478fbb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q" podUID="142bd310-9137-4db5-a5e5-27fcf478fbb5"
Oct 30 19:15:06.156932 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:06.156911    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:06.157741 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:06.157724    4924 scope.go:122] "RemoveContainer" containerID="e20c8655236127359e725e37333b40bce3b4302869e58d26f822ce947f0698d7"
Oct 30 19:15:06.157956 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:06.157923    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh_odh-ai-gateway-infra(d9aa4147-99fd-4ee4-9dd0-43ae21407002)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh" podUID="d9aa4147-99fd-4ee4-9dd0-43ae21407002"
Oct 30 19:15:06.159395 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:06.159368    4924 scope.go:122] "RemoveContainer" containerID="b9c2353176b0df07e91c4cb363b0ec4b3727b60254b491690f5277dc4523158a"
Oct 30 19:15:06.159588 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:06.159564    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96_odh-ai-gateway-infra(4ea62e28-48cf-40e4-9de4-1b767bd839ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96" podUID="4ea62e28-48cf-40e4-9de4-1b767bd839ce"
Oct 30 19:15:06.160965 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:06.160950    4924 scope.go:122] "RemoveContainer" containerID="9cef2a28ad870123b7e723f8b149aa8952716ebf53cdb97c1e5059030e447f05"
Oct 30 19:15:06.161166 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:06.161148    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg_odh-ai-gateway-infra(d9d33ee4-cc7c-4043-a179-65094f01ca5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg" podUID="d9d33ee4-cc7c-4043-a179-65094f01ca5f"
Oct 30 19:15:06.283680 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:06.282287    4924 patch_prober.go:29] interesting pod/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": read tcp 10.132.0.2:47266->10.132.0.191:15021: read: connection reset by peer" start-of-body=
Oct 30 19:15:06.283680 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:06.282342    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745" podUID="4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": read tcp 10.132.0.2:47266->10.132.0.191:15021: read: connection reset by peer"
Oct 30 19:15:06.837308 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:06.837270    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="000b4c54-e96e-497e-b489-504f2f52094c" path="/var/lib/kubelet/pods/000b4c54-e96e-497e-b489-504f2f52094c/volumes"
Oct 30 19:15:07.172558 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:07.172520    4924 generic.go:356] "Generic (PLEG): container finished" podID="4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" containerID="0c9a2365f74bd1886974b2b5ae9c42ad508a8e8a50df847095f1382fcdfb7abe" exitCode=0
Oct 30 19:15:07.172964 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:07.172595    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745" event={"ID":"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421","Type":"ContainerDied","Data":"0c9a2365f74bd1886974b2b5ae9c42ad508a8e8a50df847095f1382fcdfb7abe"}
Oct 30 19:15:07.173225 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:07.173205    4924 scope.go:122] "RemoveContainer" containerID="0c9a2365f74bd1886974b2b5ae9c42ad508a8e8a50df847095f1382fcdfb7abe"
Oct 30 19:15:07.174963 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:07.174937    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:15:07.175101 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:07.174998    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:15:07.175101 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:07.175036    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:15:08.182314 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:08.182280    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745" event={"ID":"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421","Type":"ContainerStarted","Data":"ce8ad97e0b51f7e259c8991ff62a33b5037bc613b4da58d665e6dd0d99a50e2b"}
Oct 30 19:15:08.187772 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:08.187752    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:15:08.187935 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:08.187913    4924 patch_prober.go:29] interesting pod/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 19:15:08.187985 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:08.187957    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745" podUID="4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 19:15:08.632526 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:08.632494    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:08.662076 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:08.662038    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj"]
Oct 30 19:15:08.980084 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:08.976187    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj"
Oct 30 19:15:09.049446 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:09.049408    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bb49af5-h8qkr"]
Oct 30 19:15:09.054047 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:09.054018    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bb49af5-h8qkr"]
Oct 30 19:15:09.095165 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:09.095115    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cbfab36-1855-4785-97e2-4c8a96cd9e0c-kube-api-access-bnng8\" (UniqueName: \"kubernetes.io/projected/4cbfab36-1855-4785-97e2-4c8a96cd9e0c-kube-api-access-bnng8\") pod \"4cbfab36-1855-4785-97e2-4c8a96cd9e0c\" (UID: \"4cbfab36-1855-4785-97e2-4c8a96cd9e0c\") "
Oct 30 19:15:09.095526 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:09.095497    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cbfab36-1855-4785-97e2-4c8a96cd9e0c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cbfab36-1855-4785-97e2-4c8a96cd9e0c-maas-api-service-ca\") pod \"4cbfab36-1855-4785-97e2-4c8a96cd9e0c\" (UID: \"4cbfab36-1855-4785-97e2-4c8a96cd9e0c\") "
Oct 30 19:15:09.095902 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:09.095875    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbfab36-1855-4785-97e2-4c8a96cd9e0c-maas-api-service-ca" pod "4cbfab36-1855-4785-97e2-4c8a96cd9e0c" (UID: "4cbfab36-1855-4785-97e2-4c8a96cd9e0c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:09.125087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:09.121414    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cbfab36-1855-4785-97e2-4c8a96cd9e0c-maas-api-service-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:09.135418 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:09.135381    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbfab36-1855-4785-97e2-4c8a96cd9e0c-kube-api-access-bnng8" pod "4cbfab36-1855-4785-97e2-4c8a96cd9e0c" (UID: "4cbfab36-1855-4785-97e2-4c8a96cd9e0c"). InnerVolumeSpecName "kube-api-access-bnng8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:09.188049 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:09.188013    4924 patch_prober.go:29] interesting pod/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 19:15:09.188501 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:09.188103    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745" podUID="4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 19:15:09.192309 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:09.192287    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj"
Oct 30 19:15:09.192452 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:09.192270    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj" event={"ID":"4cbfab36-1855-4785-97e2-4c8a96cd9e0c","Type":"ContainerDied","Data":"d670423121d7101627ed7c2bf53b4c9138428b2d3eabe90ae52d565416e5f8b2"}
Oct 30 19:15:09.192452 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:09.192411    4924 scope.go:122] "RemoveContainer" containerID="b5c303d5aff1f7944d240f77f7b74ecc19bf220b56a2fd847fccb9905852389b"
Oct 30 19:15:09.228275 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:09.227985    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnng8\" (UniqueName: \"kubernetes.io/projected/4cbfab36-1855-4785-97e2-4c8a96cd9e0c-kube-api-access-bnng8\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:09.353929 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:09.353890    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj"]
Oct 30 19:15:09.365861 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:09.365834    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-05565d-29889795-m5hsj"]
Oct 30 19:15:09.581143 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:09.578508    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:10.188355 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:10.188318    4924 patch_prober.go:29] interesting pod/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 19:15:10.188737 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:10.188372    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745" podUID="4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 19:15:10.227091 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:10.223479    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:10.443176 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:10.441317    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:10.561131 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:10.561037    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl" podUID="91780c8b-9c6c-47bd-aa29-f85529dcd00f" containerName="payload-processing" containerID="cri-o://46dba677a96f0162bcc060bb1b6e6e9ea26fcf8209680d68f40a353999a887c9" gracePeriod=25
Oct 30 19:15:10.659271 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:10.659238    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:10.669669 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:10.669630    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb" podUID="fbddd023-71f1-4a7b-90fb-ebfadb948013" containerName="payload-pre-processing" containerID="cri-o://2c1d1397685929f5c647a6629c558c3917f096fc62164aa25964f4749fedc28b" gracePeriod=25
Oct 30 19:15:10.858309 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:10.858268    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2466b48e-af9c-4269-94f9-1e612e01a044" path="/var/lib/kubelet/pods/2466b48e-af9c-4269-94f9-1e612e01a044/volumes"
Oct 30 19:15:10.864235 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:10.864205    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cbfab36-1855-4785-97e2-4c8a96cd9e0c" path="/var/lib/kubelet/pods/4cbfab36-1855-4785-97e2-4c8a96cd9e0c/volumes"
Oct 30 19:15:10.866595 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:10.866565    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05565d-j4bxr"]
Oct 30 19:15:10.914983 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:10.914944    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05565d-j4bxr"]
Oct 30 19:15:10.942012 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:10.941985    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl"
Oct 30 19:15:11.062934 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.062868    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91780c8b-9c6c-47bd-aa29-f85529dcd00f-kube-api-access-xkcrx\" (UniqueName: \"kubernetes.io/projected/91780c8b-9c6c-47bd-aa29-f85529dcd00f-kube-api-access-xkcrx\") pod \"91780c8b-9c6c-47bd-aa29-f85529dcd00f\" (UID: \"91780c8b-9c6c-47bd-aa29-f85529dcd00f\") "
Oct 30 19:15:11.063135 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.063048    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91780c8b-9c6c-47bd-aa29-f85529dcd00f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91780c8b-9c6c-47bd-aa29-f85529dcd00f-plugins-config-volume\") pod \"91780c8b-9c6c-47bd-aa29-f85529dcd00f\" (UID: \"91780c8b-9c6c-47bd-aa29-f85529dcd00f\") "
Oct 30 19:15:11.063510 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.063483    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91780c8b-9c6c-47bd-aa29-f85529dcd00f-plugins-config-volume" pod "91780c8b-9c6c-47bd-aa29-f85529dcd00f" (UID: "91780c8b-9c6c-47bd-aa29-f85529dcd00f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:11.063746 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.063718    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91780c8b-9c6c-47bd-aa29-f85529dcd00f-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:11.067033 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.067008    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91780c8b-9c6c-47bd-aa29-f85529dcd00f-kube-api-access-xkcrx" pod "91780c8b-9c6c-47bd-aa29-f85529dcd00f" (UID: "91780c8b-9c6c-47bd-aa29-f85529dcd00f"). InnerVolumeSpecName "kube-api-access-xkcrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:11.075903 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.075882    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb"
Oct 30 19:15:11.165039 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.165005    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkcrx\" (UniqueName: \"kubernetes.io/projected/91780c8b-9c6c-47bd-aa29-f85529dcd00f-kube-api-access-xkcrx\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:11.188530 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.188501    4924 patch_prober.go:29] interesting pod/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 19:15:11.188908 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.188548    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745" podUID="4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 19:15:11.233687 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.233643    4924 generic.go:356] "Generic (PLEG): container finished" podID="91780c8b-9c6c-47bd-aa29-f85529dcd00f" containerID="46dba677a96f0162bcc060bb1b6e6e9ea26fcf8209680d68f40a353999a887c9" exitCode=0
Oct 30 19:15:11.233886 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.233711    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl"
Oct 30 19:15:11.233886 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.233731    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl" event={"ID":"91780c8b-9c6c-47bd-aa29-f85529dcd00f","Type":"ContainerDied","Data":"46dba677a96f0162bcc060bb1b6e6e9ea26fcf8209680d68f40a353999a887c9"}
Oct 30 19:15:11.233886 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.233806    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl" event={"ID":"91780c8b-9c6c-47bd-aa29-f85529dcd00f","Type":"ContainerDied","Data":"fe9dbb022c6a861852b8bc523a64f567b7b72bef4693f6c1835a247a6c3ba9e1"}
Oct 30 19:15:11.233886 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.233842    4924 scope.go:122] "RemoveContainer" containerID="46dba677a96f0162bcc060bb1b6e6e9ea26fcf8209680d68f40a353999a887c9"
Oct 30 19:15:11.239390 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.239364    4924 generic.go:356] "Generic (PLEG): container finished" podID="fbddd023-71f1-4a7b-90fb-ebfadb948013" containerID="2c1d1397685929f5c647a6629c558c3917f096fc62164aa25964f4749fedc28b" exitCode=0
Oct 30 19:15:11.239517 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.239416    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb" event={"ID":"fbddd023-71f1-4a7b-90fb-ebfadb948013","Type":"ContainerDied","Data":"2c1d1397685929f5c647a6629c558c3917f096fc62164aa25964f4749fedc28b"}
Oct 30 19:15:11.239517 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.239423    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb"
Oct 30 19:15:11.239517 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.239462    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb" event={"ID":"fbddd023-71f1-4a7b-90fb-ebfadb948013","Type":"ContainerDied","Data":"470fee6f5099718bb0c5158aa981f28f3dcc9bfb36d677b7866a2de6bf29e58b"}
Oct 30 19:15:11.256404 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.256382    4924 scope.go:122] "RemoveContainer" containerID="46dba677a96f0162bcc060bb1b6e6e9ea26fcf8209680d68f40a353999a887c9"
Oct 30 19:15:11.256914 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:11.256890    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46dba677a96f0162bcc060bb1b6e6e9ea26fcf8209680d68f40a353999a887c9\": container with ID starting with 46dba677a96f0162bcc060bb1b6e6e9ea26fcf8209680d68f40a353999a887c9 not found: ID does not exist" containerID="46dba677a96f0162bcc060bb1b6e6e9ea26fcf8209680d68f40a353999a887c9"
Oct 30 19:15:11.257010 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.256922    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46dba677a96f0162bcc060bb1b6e6e9ea26fcf8209680d68f40a353999a887c9"} err="failed to get container status \"46dba677a96f0162bcc060bb1b6e6e9ea26fcf8209680d68f40a353999a887c9\": rpc error: code = NotFound desc = could not find container \"46dba677a96f0162bcc060bb1b6e6e9ea26fcf8209680d68f40a353999a887c9\": container with ID starting with 46dba677a96f0162bcc060bb1b6e6e9ea26fcf8209680d68f40a353999a887c9 not found: ID does not exist"
Oct 30 19:15:11.257010 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.256940    4924 scope.go:122] "RemoveContainer" containerID="2c1d1397685929f5c647a6629c558c3917f096fc62164aa25964f4749fedc28b"
Oct 30 19:15:11.265840 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.265816    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbddd023-71f1-4a7b-90fb-ebfadb948013-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbddd023-71f1-4a7b-90fb-ebfadb948013-plugins-config-volume\") pod \"fbddd023-71f1-4a7b-90fb-ebfadb948013\" (UID: \"fbddd023-71f1-4a7b-90fb-ebfadb948013\") "
Oct 30 19:15:11.265999 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.265965    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbddd023-71f1-4a7b-90fb-ebfadb948013-kube-api-access-bdqzr\" (UniqueName: \"kubernetes.io/projected/fbddd023-71f1-4a7b-90fb-ebfadb948013-kube-api-access-bdqzr\") pod \"fbddd023-71f1-4a7b-90fb-ebfadb948013\" (UID: \"fbddd023-71f1-4a7b-90fb-ebfadb948013\") "
Oct 30 19:15:11.266204 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.266181    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbddd023-71f1-4a7b-90fb-ebfadb948013-plugins-config-volume" pod "fbddd023-71f1-4a7b-90fb-ebfadb948013" (UID: "fbddd023-71f1-4a7b-90fb-ebfadb948013"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:11.266498 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.266478    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbddd023-71f1-4a7b-90fb-ebfadb948013-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:11.266916 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.266900    4924 scope.go:122] "RemoveContainer" containerID="2c1d1397685929f5c647a6629c558c3917f096fc62164aa25964f4749fedc28b"
Oct 30 19:15:11.267249 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:11.267219    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c1d1397685929f5c647a6629c558c3917f096fc62164aa25964f4749fedc28b\": container with ID starting with 2c1d1397685929f5c647a6629c558c3917f096fc62164aa25964f4749fedc28b not found: ID does not exist" containerID="2c1d1397685929f5c647a6629c558c3917f096fc62164aa25964f4749fedc28b"
Oct 30 19:15:11.267334 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.267250    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c1d1397685929f5c647a6629c558c3917f096fc62164aa25964f4749fedc28b"} err="failed to get container status \"2c1d1397685929f5c647a6629c558c3917f096fc62164aa25964f4749fedc28b\": rpc error: code = NotFound desc = could not find container \"2c1d1397685929f5c647a6629c558c3917f096fc62164aa25964f4749fedc28b\": container with ID starting with 2c1d1397685929f5c647a6629c558c3917f096fc62164aa25964f4749fedc28b not found: ID does not exist"
Oct 30 19:15:11.267975 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.267955    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbddd023-71f1-4a7b-90fb-ebfadb948013-kube-api-access-bdqzr" pod "fbddd023-71f1-4a7b-90fb-ebfadb948013" (UID: "fbddd023-71f1-4a7b-90fb-ebfadb948013"). InnerVolumeSpecName "kube-api-access-bdqzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:11.327922 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.327834    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:11.368117 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.368058    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdqzr\" (UniqueName: \"kubernetes.io/projected/fbddd023-71f1-4a7b-90fb-ebfadb948013-kube-api-access-bdqzr\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:11.443492 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.443455    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl"]
Oct 30 19:15:11.450574 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.450547    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-314bd6fa-77d6858c46-dwnbl"]
Oct 30 19:15:11.658769 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.658738    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb"]
Oct 30 19:15:11.663468 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.663443    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-314bd6fa-7c6745c447-r8frb"]
Oct 30 19:15:11.739268 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.739234    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:11.929660 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:11.929621    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:12.190145 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.189420    4924 patch_prober.go:29] interesting pod/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 19:15:12.190145 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.189499    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745" podUID="4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 19:15:12.220091 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.220006    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:12.590579 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.590493    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:12.838556 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.838521    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6000b1cb-428d-4beb-b0e5-3619a6a61096" path="/var/lib/kubelet/pods/6000b1cb-428d-4beb-b0e5-3619a6a61096/volumes"
Oct 30 19:15:12.839009 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.838994    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91780c8b-9c6c-47bd-aa29-f85529dcd00f" path="/var/lib/kubelet/pods/91780c8b-9c6c-47bd-aa29-f85529dcd00f/volumes"
Oct 30 19:15:12.839450 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.839435    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbddd023-71f1-4a7b-90fb-ebfadb948013" path="/var/lib/kubelet/pods/fbddd023-71f1-4a7b-90fb-ebfadb948013/volumes"
Oct 30 19:15:12.872889 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.872865    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87"]
Oct 30 19:15:12.873701 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.873681    4924 cpu_manager.go:414] "containerMap: removing container" podUID="4cbfab36-1855-4785-97e2-4c8a96cd9e0c" containerName="cleanup"
Oct 30 19:15:12.873798 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.873703    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cbfab36-1855-4785-97e2-4c8a96cd9e0c" containerName="cleanup"
Oct 30 19:15:12.873798 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.873746    4924 cpu_manager.go:414] "containerMap: removing container" podUID="fbddd023-71f1-4a7b-90fb-ebfadb948013" containerName="payload-pre-processing"
Oct 30 19:15:12.873798 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.873752    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbddd023-71f1-4a7b-90fb-ebfadb948013" containerName="payload-pre-processing"
Oct 30 19:15:12.873798 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.873763    4924 cpu_manager.go:414] "containerMap: removing container" podUID="000b4c54-e96e-497e-b489-504f2f52094c" containerName="cleanup"
Oct 30 19:15:12.873798 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.873769    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="000b4c54-e96e-497e-b489-504f2f52094c" containerName="cleanup"
Oct 30 19:15:12.873798 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.873780    4924 cpu_manager.go:414] "containerMap: removing container" podUID="4cbfab36-1855-4785-97e2-4c8a96cd9e0c" containerName="cleanup"
Oct 30 19:15:12.873798 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.873786    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cbfab36-1855-4785-97e2-4c8a96cd9e0c" containerName="cleanup"
Oct 30 19:15:12.874035 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.873809    4924 cpu_manager.go:414] "containerMap: removing container" podUID="91780c8b-9c6c-47bd-aa29-f85529dcd00f" containerName="payload-processing"
Oct 30 19:15:12.874035 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.873814    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91780c8b-9c6c-47bd-aa29-f85529dcd00f" containerName="payload-processing"
Oct 30 19:15:12.874035 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.873933    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="91780c8b-9c6c-47bd-aa29-f85529dcd00f" containerName="payload-processing"
Oct 30 19:15:12.874035 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.873941    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="000b4c54-e96e-497e-b489-504f2f52094c" containerName="cleanup"
Oct 30 19:15:12.874035 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.873949    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cbfab36-1855-4785-97e2-4c8a96cd9e0c" containerName="cleanup"
Oct 30 19:15:12.874035 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.873958    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbddd023-71f1-4a7b-90fb-ebfadb948013" containerName="payload-pre-processing"
Oct 30 19:15:12.874035 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.873967    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cbfab36-1855-4785-97e2-4c8a96cd9e0c" containerName="cleanup"
Oct 30 19:15:12.879439 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.879416    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87"
Oct 30 19:15:12.892611 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.892586    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwnlp\" (UniqueName: \"kubernetes.io/projected/51b9a4f5-d2e7-462e-ab62-03ef1a125d89-kube-api-access-vwnlp\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87\" (UID: \"51b9a4f5-d2e7-462e-ab62-03ef1a125d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87"
Oct 30 19:15:12.892793 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.892775    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b9a4f5-d2e7-462e-ab62-03ef1a125d89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87\" (UID: \"51b9a4f5-d2e7-462e-ab62-03ef1a125d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87"
Oct 30 19:15:12.892887 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.892872    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b9a4f5-d2e7-462e-ab62-03ef1a125d89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87\" (UID: \"51b9a4f5-d2e7-462e-ab62-03ef1a125d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87"
Oct 30 19:15:12.935434 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.935401    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87"]
Oct 30 19:15:12.975965 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.975938    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:15:12.994502 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.994475    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwnlp\" (UniqueName: \"kubernetes.io/projected/51b9a4f5-d2e7-462e-ab62-03ef1a125d89-kube-api-access-vwnlp\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87\" (UID: \"51b9a4f5-d2e7-462e-ab62-03ef1a125d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87"
Oct 30 19:15:12.994645 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.994597    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b9a4f5-d2e7-462e-ab62-03ef1a125d89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87\" (UID: \"51b9a4f5-d2e7-462e-ab62-03ef1a125d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87"
Oct 30 19:15:12.994704 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.994688    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b9a4f5-d2e7-462e-ab62-03ef1a125d89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87\" (UID: \"51b9a4f5-d2e7-462e-ab62-03ef1a125d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87"
Oct 30 19:15:12.995383 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.995292    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b9a4f5-d2e7-462e-ab62-03ef1a125d89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87\" (UID: \"51b9a4f5-d2e7-462e-ab62-03ef1a125d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87"
Oct 30 19:15:12.997677 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:12.997649    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b9a4f5-d2e7-462e-ab62-03ef1a125d89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87\" (UID: \"51b9a4f5-d2e7-462e-ab62-03ef1a125d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87"
Oct 30 19:15:13.049482 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:13.049441    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwnlp\" (UniqueName: \"kubernetes.io/projected/51b9a4f5-d2e7-462e-ab62-03ef1a125d89-kube-api-access-vwnlp\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87\" (UID: \"51b9a4f5-d2e7-462e-ab62-03ef1a125d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87"
Oct 30 19:15:13.188256 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:13.188178    4924 patch_prober.go:29] interesting pod/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 19:15:13.188256 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:13.188241    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745" podUID="4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 19:15:13.199640 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:13.199617    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87"
Oct 30 19:15:13.377302 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:13.377261    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc" podUID="0b676034-384c-4c43-a8f5-991977f109ce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:15:13.614286 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:13.614258    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87"]
Oct 30 19:15:13.660683 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:13.660646    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc"
Oct 30 19:15:13.706339 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:13.706311    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b676034-384c-4c43-a8f5-991977f109ce-kube-api-access-p8m44\" (UniqueName: \"kubernetes.io/projected/0b676034-384c-4c43-a8f5-991977f109ce-kube-api-access-p8m44\") pod \"0b676034-384c-4c43-a8f5-991977f109ce\" (UID: \"0b676034-384c-4c43-a8f5-991977f109ce\") "
Oct 30 19:15:13.706513 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:13.706428    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b676034-384c-4c43-a8f5-991977f109ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b676034-384c-4c43-a8f5-991977f109ce-plugins-config-volume\") pod \"0b676034-384c-4c43-a8f5-991977f109ce\" (UID: \"0b676034-384c-4c43-a8f5-991977f109ce\") "
Oct 30 19:15:13.706879 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:13.706850    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b676034-384c-4c43-a8f5-991977f109ce-plugins-config-volume" pod "0b676034-384c-4c43-a8f5-991977f109ce" (UID: "0b676034-384c-4c43-a8f5-991977f109ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:13.707174 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:13.707158    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b676034-384c-4c43-a8f5-991977f109ce-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:13.708474 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:13.708452    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b676034-384c-4c43-a8f5-991977f109ce-kube-api-access-p8m44" pod "0b676034-384c-4c43-a8f5-991977f109ce" (UID: "0b676034-384c-4c43-a8f5-991977f109ce"). InnerVolumeSpecName "kube-api-access-p8m44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:13.808437 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:13.808366    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8m44\" (UniqueName: \"kubernetes.io/projected/0b676034-384c-4c43-a8f5-991977f109ce-kube-api-access-p8m44\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:14.191899 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:14.191869    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:15:14.192261 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:14.192236    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:15:14.192974 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:14.192955    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:15:14.287133 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:14.287101    4924 generic.go:356] "Generic (PLEG): container finished" podID="0b676034-384c-4c43-a8f5-991977f109ce" containerID="83824a1dbc82e10702511263b7f2b315f237370d8490afc2746a66aa7a51ffb2" exitCode=0
Oct 30 19:15:14.287599 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:14.287194    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc" event={"ID":"0b676034-384c-4c43-a8f5-991977f109ce","Type":"ContainerDied","Data":"83824a1dbc82e10702511263b7f2b315f237370d8490afc2746a66aa7a51ffb2"}
Oct 30 19:15:14.287599 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:14.287210    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc"
Oct 30 19:15:14.287599 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:14.287241    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc" event={"ID":"0b676034-384c-4c43-a8f5-991977f109ce","Type":"ContainerDied","Data":"c1f1446b8d7eeaf49ff8f16bcfc7a2a5de51ec2132a0298f49e68389df5de514"}
Oct 30 19:15:14.287599 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:14.287269    4924 scope.go:122] "RemoveContainer" containerID="83824a1dbc82e10702511263b7f2b315f237370d8490afc2746a66aa7a51ffb2"
Oct 30 19:15:14.289304 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:14.289281    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87" event={"ID":"51b9a4f5-d2e7-462e-ab62-03ef1a125d89","Type":"ContainerStarted","Data":"289433759f06be19a725938fb04d2bc9b1091b5e234674be23844d24a5cdc034"}
Oct 30 19:15:14.289414 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:14.289324    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87" event={"ID":"51b9a4f5-d2e7-462e-ab62-03ef1a125d89","Type":"ContainerStarted","Data":"3cb51037de7d7e3b7b54523b59cfbba33a5f7d64d94ab0c5fa3e70a1c9764e82"}
Oct 30 19:15:14.298243 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:14.298223    4924 scope.go:122] "RemoveContainer" containerID="83824a1dbc82e10702511263b7f2b315f237370d8490afc2746a66aa7a51ffb2"
Oct 30 19:15:14.298510 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:14.298484    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83824a1dbc82e10702511263b7f2b315f237370d8490afc2746a66aa7a51ffb2\": container with ID starting with 83824a1dbc82e10702511263b7f2b315f237370d8490afc2746a66aa7a51ffb2 not found: ID does not exist" containerID="83824a1dbc82e10702511263b7f2b315f237370d8490afc2746a66aa7a51ffb2"
Oct 30 19:15:14.298556 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:14.298510    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83824a1dbc82e10702511263b7f2b315f237370d8490afc2746a66aa7a51ffb2"} err="failed to get container status \"83824a1dbc82e10702511263b7f2b315f237370d8490afc2746a66aa7a51ffb2\": rpc error: code = NotFound desc = could not find container \"83824a1dbc82e10702511263b7f2b315f237370d8490afc2746a66aa7a51ffb2\": container with ID starting with 83824a1dbc82e10702511263b7f2b315f237370d8490afc2746a66aa7a51ffb2 not found: ID does not exist"
Oct 30 19:15:14.416330 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:14.416284    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87" podStartSLOduration=2.416271478 podStartE2EDuration="2.416271478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:14.413976028 +0000 UTC m=+2315.691026473" watchObservedRunningTime="2026-10-30 19:15:14.416271478 +0000 UTC m=+2315.693321922"
Oct 30 19:15:14.524166 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:14.524088    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc"]
Oct 30 19:15:14.529287 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:14.529260    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-649d16-5984655f58-v9njc"]
Oct 30 19:15:14.827767 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:14.827690    4924 scope.go:122] "RemoveContainer" containerID="eb717f40124cc0f021391ed892d2f183131db2b021b55423dfb41d591904158c"
Oct 30 19:15:14.836421 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:14.836395    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b676034-384c-4c43-a8f5-991977f109ce" path="/var/lib/kubelet/pods/0b676034-384c-4c43-a8f5-991977f109ce/volumes"
Oct 30 19:15:15.298736 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:15.298701    4924 generic.go:356] "Generic (PLEG): container finished" podID="51b9a4f5-d2e7-462e-ab62-03ef1a125d89" containerID="289433759f06be19a725938fb04d2bc9b1091b5e234674be23844d24a5cdc034" exitCode=0
Oct 30 19:15:15.299164 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:15.298754    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87" event={"ID":"51b9a4f5-d2e7-462e-ab62-03ef1a125d89","Type":"ContainerDied","Data":"289433759f06be19a725938fb04d2bc9b1091b5e234674be23844d24a5cdc034"}
Oct 30 19:15:15.300718 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:15.300698    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf" event={"ID":"7f9d64eb-f439-47cc-8f2c-d7eaf03fd573","Type":"ContainerStarted","Data":"31a49e79224a64b51630d013a661098237a1668ec17269f57ebfac0aebc70cd4"}
Oct 30 19:15:15.712833 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:15.712798    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:15:15.827923 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:15.827888    4924 scope.go:122] "RemoveContainer" containerID="b9c2353176b0df07e91c4cb363b0ec4b3727b60254b491690f5277dc4523158a"
Oct 30 19:15:15.845944 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:15.845910    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:16.311238 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:16.311202    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96" event={"ID":"4ea62e28-48cf-40e4-9de4-1b767bd839ce","Type":"ContainerStarted","Data":"7cc084c75a58c0dd5d877b9000b5d1f08f4f3d6055053ea3391c84f0cb64c654"}
Oct 30 19:15:16.313167 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:16.313143    4924 generic.go:356] "Generic (PLEG): container finished" podID="7f9d64eb-f439-47cc-8f2c-d7eaf03fd573" containerID="31a49e79224a64b51630d013a661098237a1668ec17269f57ebfac0aebc70cd4" exitCode=60
Oct 30 19:15:16.313295 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:16.313217    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf" event={"ID":"7f9d64eb-f439-47cc-8f2c-d7eaf03fd573","Type":"ContainerDied","Data":"31a49e79224a64b51630d013a661098237a1668ec17269f57ebfac0aebc70cd4"}
Oct 30 19:15:16.313295 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:16.313273    4924 scope.go:122] "RemoveContainer" containerID="eb717f40124cc0f021391ed892d2f183131db2b021b55423dfb41d591904158c"
Oct 30 19:15:16.313684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:16.313671    4924 scope.go:122] "RemoveContainer" containerID="31a49e79224a64b51630d013a661098237a1668ec17269f57ebfac0aebc70cd4"
Oct 30 19:15:16.314050 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:16.314032    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf_odh-ai-gateway-infra(7f9d64eb-f439-47cc-8f2c-d7eaf03fd573)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf" podUID="7f9d64eb-f439-47cc-8f2c-d7eaf03fd573"
Oct 30 19:15:16.480713 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:16.480651    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87"
Oct 30 19:15:16.526098 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:16.526058    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf"]
Oct 30 19:15:16.556507 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:16.556481    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51b9a4f5-d2e7-462e-ab62-03ef1a125d89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b9a4f5-d2e7-462e-ab62-03ef1a125d89-maas-api-openshift-ca\") pod \"51b9a4f5-d2e7-462e-ab62-03ef1a125d89\" (UID: \"51b9a4f5-d2e7-462e-ab62-03ef1a125d89\") "
Oct 30 19:15:16.556682 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:16.556583    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51b9a4f5-d2e7-462e-ab62-03ef1a125d89-kube-api-access-vwnlp\" (UniqueName: \"kubernetes.io/projected/51b9a4f5-d2e7-462e-ab62-03ef1a125d89-kube-api-access-vwnlp\") pod \"51b9a4f5-d2e7-462e-ab62-03ef1a125d89\" (UID: \"51b9a4f5-d2e7-462e-ab62-03ef1a125d89\") "
Oct 30 19:15:16.556682 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:16.556623    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51b9a4f5-d2e7-462e-ab62-03ef1a125d89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b9a4f5-d2e7-462e-ab62-03ef1a125d89-maas-api-xks-ca\") pod \"51b9a4f5-d2e7-462e-ab62-03ef1a125d89\" (UID: \"51b9a4f5-d2e7-462e-ab62-03ef1a125d89\") "
Oct 30 19:15:16.556875 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:16.556853    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b9a4f5-d2e7-462e-ab62-03ef1a125d89-maas-api-openshift-ca" pod "51b9a4f5-d2e7-462e-ab62-03ef1a125d89" (UID: "51b9a4f5-d2e7-462e-ab62-03ef1a125d89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:16.557200 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:16.557177    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b9a4f5-d2e7-462e-ab62-03ef1a125d89-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:16.558687 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:16.558662    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b9a4f5-d2e7-462e-ab62-03ef1a125d89-kube-api-access-vwnlp" pod "51b9a4f5-d2e7-462e-ab62-03ef1a125d89" (UID: "51b9a4f5-d2e7-462e-ab62-03ef1a125d89"). InnerVolumeSpecName "kube-api-access-vwnlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:16.558687 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:16.558679    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51b9a4f5-d2e7-462e-ab62-03ef1a125d89-maas-api-xks-ca" pod "51b9a4f5-d2e7-462e-ab62-03ef1a125d89" (UID: "51b9a4f5-d2e7-462e-ab62-03ef1a125d89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:16.658483 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:16.658444    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b9a4f5-d2e7-462e-ab62-03ef1a125d89-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:16.658658 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:16.658488    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwnlp\" (UniqueName: \"kubernetes.io/projected/51b9a4f5-d2e7-462e-ab62-03ef1a125d89-kube-api-access-vwnlp\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:16.827610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:16.827524    4924 scope.go:122] "RemoveContainer" containerID="9cef2a28ad870123b7e723f8b149aa8952716ebf53cdb97c1e5059030e447f05"
Oct 30 19:15:17.166427 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:17.166383    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:17.323385 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:17.323354    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87"
Oct 30 19:15:17.323385 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:17.323372    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87" event={"ID":"51b9a4f5-d2e7-462e-ab62-03ef1a125d89","Type":"ContainerDied","Data":"3cb51037de7d7e3b7b54523b59cfbba33a5f7d64d94ab0c5fa3e70a1c9764e82"}
Oct 30 19:15:17.323858 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:17.323455    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cb51037de7d7e3b7b54523b59cfbba33a5f7d64d94ab0c5fa3e70a1c9764e82"
Oct 30 19:15:17.325509 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:17.325437    4924 generic.go:356] "Generic (PLEG): container finished" podID="4ea62e28-48cf-40e4-9de4-1b767bd839ce" containerID="7cc084c75a58c0dd5d877b9000b5d1f08f4f3d6055053ea3391c84f0cb64c654" exitCode=60
Oct 30 19:15:17.325710 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:17.325517    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96" event={"ID":"4ea62e28-48cf-40e4-9de4-1b767bd839ce","Type":"ContainerDied","Data":"7cc084c75a58c0dd5d877b9000b5d1f08f4f3d6055053ea3391c84f0cb64c654"}
Oct 30 19:15:17.325710 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:17.325593    4924 scope.go:122] "RemoveContainer" containerID="b9c2353176b0df07e91c4cb363b0ec4b3727b60254b491690f5277dc4523158a"
Oct 30 19:15:17.326093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:17.326058    4924 scope.go:122] "RemoveContainer" containerID="7cc084c75a58c0dd5d877b9000b5d1f08f4f3d6055053ea3391c84f0cb64c654"
Oct 30 19:15:17.326436 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:17.326415    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96_odh-ai-gateway-infra(4ea62e28-48cf-40e4-9de4-1b767bd839ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96" podUID="4ea62e28-48cf-40e4-9de4-1b767bd839ce"
Oct 30 19:15:17.327848 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:17.327826    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg" event={"ID":"d9d33ee4-cc7c-4043-a179-65094f01ca5f","Type":"ContainerStarted","Data":"4dceb8745a73ad7bd37294ff7c0b41131be1d0498ef5c741a8d869a688f0e5e5"}
Oct 30 19:15:17.373257 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:17.373209    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:17.433102 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:17.432995    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96"]
Oct 30 19:15:17.491024 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:17.490994    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf"
Oct 30 19:15:17.498782 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:17.498753    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"]
Oct 30 19:15:17.532555 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:17.532526    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"]
Oct 30 19:15:17.572597 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:17.572572    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f9d64eb-f439-47cc-8f2c-d7eaf03fd573-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f9d64eb-f439-47cc-8f2c-d7eaf03fd573-maas-api-service-ca\") pod \"7f9d64eb-f439-47cc-8f2c-d7eaf03fd573\" (UID: \"7f9d64eb-f439-47cc-8f2c-d7eaf03fd573\") "
Oct 30 19:15:17.572775 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:17.572632    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f9d64eb-f439-47cc-8f2c-d7eaf03fd573-kube-api-access-5nqv9\" (UniqueName: \"kubernetes.io/projected/7f9d64eb-f439-47cc-8f2c-d7eaf03fd573-kube-api-access-5nqv9\") pod \"7f9d64eb-f439-47cc-8f2c-d7eaf03fd573\" (UID: \"7f9d64eb-f439-47cc-8f2c-d7eaf03fd573\") "
Oct 30 19:15:17.572993 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:17.572971    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f9d64eb-f439-47cc-8f2c-d7eaf03fd573-maas-api-service-ca" pod "7f9d64eb-f439-47cc-8f2c-d7eaf03fd573" (UID: "7f9d64eb-f439-47cc-8f2c-d7eaf03fd573"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:17.573368 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:17.573351    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f9d64eb-f439-47cc-8f2c-d7eaf03fd573-maas-api-service-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:17.574657 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:17.574638    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f9d64eb-f439-47cc-8f2c-d7eaf03fd573-kube-api-access-5nqv9" pod "7f9d64eb-f439-47cc-8f2c-d7eaf03fd573" (UID: "7f9d64eb-f439-47cc-8f2c-d7eaf03fd573"). InnerVolumeSpecName "kube-api-access-5nqv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:17.659561 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:17.659525    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:17.675183 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:17.675153    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nqv9\" (UniqueName: \"kubernetes.io/projected/7f9d64eb-f439-47cc-8f2c-d7eaf03fd573-kube-api-access-5nqv9\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:17.828147 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:17.828113    4924 scope.go:122] "RemoveContainer" containerID="e20c8655236127359e725e37333b40bce3b4302869e58d26f822ce947f0698d7"
Oct 30 19:15:17.828364 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:17.828252    4924 scope.go:122] "RemoveContainer" containerID="f6992130696f4be8093f28f8409e9eac848b44c35738d787c01c332a251c9c70"
Oct 30 19:15:17.828455 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:17.828365    4924 scope.go:122] "RemoveContainer" containerID="e880e44d5bd8cf65996036201caa9aff1e994be589d9fcb42526397c3c9d72a5"
Oct 30 19:15:17.828513 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:17.828449    4924 scope.go:122] "RemoveContainer" containerID="689804f1c62420b6997dd80a71df8531bf7e6d9c8804f87f7f689ad0d93834b7"
Oct 30 19:15:18.342266 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:18.342237    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx" event={"ID":"70bd8cf2-4a28-46e2-8223-27e0890f7444","Type":"ContainerStarted","Data":"f6d0dbd4d9221789d4645a3432b03191aa45b6bd2365d678cd7959ccd7dd2c65"}
Oct 30 19:15:18.343662 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:18.343637    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf"
Oct 30 19:15:18.343662 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:18.343647    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf" event={"ID":"7f9d64eb-f439-47cc-8f2c-d7eaf03fd573","Type":"ContainerDied","Data":"8e9888539de7f386f3b8dcf9fc731b306fad510a230fd8f156909c63d7372ff4"}
Oct 30 19:15:18.343826 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:18.343701    4924 scope.go:122] "RemoveContainer" containerID="31a49e79224a64b51630d013a661098237a1668ec17269f57ebfac0aebc70cd4"
Oct 30 19:15:18.345865 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:18.345831    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g" event={"ID":"534a63d0-c4b9-48cb-88f0-bed24a366017","Type":"ContainerStarted","Data":"47ef9a29fc3f3767a342eab9064f71a2693591511a4e8ffcfb8f7437beaf5c38"}
Oct 30 19:15:18.349835 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:18.349810    4924 generic.go:356] "Generic (PLEG): container finished" podID="d9d33ee4-cc7c-4043-a179-65094f01ca5f" containerID="4dceb8745a73ad7bd37294ff7c0b41131be1d0498ef5c741a8d869a688f0e5e5" exitCode=60
Oct 30 19:15:18.349925 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:18.349843    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg" event={"ID":"d9d33ee4-cc7c-4043-a179-65094f01ca5f","Type":"ContainerDied","Data":"4dceb8745a73ad7bd37294ff7c0b41131be1d0498ef5c741a8d869a688f0e5e5"}
Oct 30 19:15:18.350197 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:18.350181    4924 scope.go:122] "RemoveContainer" containerID="4dceb8745a73ad7bd37294ff7c0b41131be1d0498ef5c741a8d869a688f0e5e5"
Oct 30 19:15:18.350426 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:18.350409    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg_odh-ai-gateway-infra(d9d33ee4-cc7c-4043-a179-65094f01ca5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg" podUID="d9d33ee4-cc7c-4043-a179-65094f01ca5f"
Oct 30 19:15:18.355907 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:18.355888    4924 scope.go:122] "RemoveContainer" containerID="9cef2a28ad870123b7e723f8b149aa8952716ebf53cdb97c1e5059030e447f05"
Oct 30 19:15:18.508511 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:18.508489    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96"
Oct 30 19:15:18.586997 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:18.586948    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea62e28-48cf-40e4-9de4-1b767bd839ce-kube-api-access-8qps2\" (UniqueName: \"kubernetes.io/projected/4ea62e28-48cf-40e4-9de4-1b767bd839ce-kube-api-access-8qps2\") pod \"4ea62e28-48cf-40e4-9de4-1b767bd839ce\" (UID: \"4ea62e28-48cf-40e4-9de4-1b767bd839ce\") "
Oct 30 19:15:18.587264 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:18.587194    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ea62e28-48cf-40e4-9de4-1b767bd839ce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea62e28-48cf-40e4-9de4-1b767bd839ce-maas-api-service-ca\") pod \"4ea62e28-48cf-40e4-9de4-1b767bd839ce\" (UID: \"4ea62e28-48cf-40e4-9de4-1b767bd839ce\") "
Oct 30 19:15:18.587713 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:18.587687    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea62e28-48cf-40e4-9de4-1b767bd839ce-maas-api-service-ca" pod "4ea62e28-48cf-40e4-9de4-1b767bd839ce" (UID: "4ea62e28-48cf-40e4-9de4-1b767bd839ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:18.589656 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:18.589632    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea62e28-48cf-40e4-9de4-1b767bd839ce-kube-api-access-8qps2" pod "4ea62e28-48cf-40e4-9de4-1b767bd839ce" (UID: "4ea62e28-48cf-40e4-9de4-1b767bd839ce"). InnerVolumeSpecName "kube-api-access-8qps2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:18.625179 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:18.625110    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg"]
Oct 30 19:15:18.689163 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:18.689133    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qps2\" (UniqueName: \"kubernetes.io/projected/4ea62e28-48cf-40e4-9de4-1b767bd839ce-kube-api-access-8qps2\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:18.689163 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:18.689160    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea62e28-48cf-40e4-9de4-1b767bd839ce-maas-api-service-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:18.772054 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:18.772014    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf"]
Oct 30 19:15:18.775698 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:18.775671    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f80ac-29889795-rpjlf"]
Oct 30 19:15:18.827267 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:18.827245    4924 scope.go:122] "RemoveContainer" containerID="8abffc49dc96cbf26716d2ae6d38998cdd29acb4dd1fd8ad423403b31b579b6d"
Oct 30 19:15:18.836511 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:18.836487    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f9d64eb-f439-47cc-8f2c-d7eaf03fd573" path="/var/lib/kubelet/pods/7f9d64eb-f439-47cc-8f2c-d7eaf03fd573/volumes"
Oct 30 19:15:19.360076 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.360040    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q" event={"ID":"142bd310-9137-4db5-a5e5-27fcf478fbb5","Type":"ContainerStarted","Data":"1afc3451bd896234f515a099df9a75b0fbf1002797d835261775612352235d1a"}
Oct 30 19:15:19.362057 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.362035    4924 generic.go:356] "Generic (PLEG): container finished" podID="d9aa4147-99fd-4ee4-9dd0-43ae21407002" containerID="ec53e51287de5c802198e31d39277109202a4f9142b30f8eaa838ef369266049" exitCode=60
Oct 30 19:15:19.362202 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.362108    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh" event={"ID":"d9aa4147-99fd-4ee4-9dd0-43ae21407002","Type":"ContainerDied","Data":"ec53e51287de5c802198e31d39277109202a4f9142b30f8eaa838ef369266049"}
Oct 30 19:15:19.362202 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.362153    4924 scope.go:122] "RemoveContainer" containerID="e20c8655236127359e725e37333b40bce3b4302869e58d26f822ce947f0698d7"
Oct 30 19:15:19.362509 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.362493    4924 scope.go:122] "RemoveContainer" containerID="ec53e51287de5c802198e31d39277109202a4f9142b30f8eaa838ef369266049"
Oct 30 19:15:19.362844 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:19.362823    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh_odh-ai-gateway-infra(d9aa4147-99fd-4ee4-9dd0-43ae21407002)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh" podUID="d9aa4147-99fd-4ee4-9dd0-43ae21407002"
Oct 30 19:15:19.363624 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.363605    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96" event={"ID":"4ea62e28-48cf-40e4-9de4-1b767bd839ce","Type":"ContainerDied","Data":"e75a7e90b8abfee55f4943f7a7e33251527e35310eaf24556f3f1b1b89f91818"}
Oct 30 19:15:19.363624 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.363619    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96"
Oct 30 19:15:19.367676 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.367655    4924 generic.go:356] "Generic (PLEG): container finished" podID="70bd8cf2-4a28-46e2-8223-27e0890f7444" containerID="f6d0dbd4d9221789d4645a3432b03191aa45b6bd2365d678cd7959ccd7dd2c65" exitCode=60
Oct 30 19:15:19.367780 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.367732    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx" event={"ID":"70bd8cf2-4a28-46e2-8223-27e0890f7444","Type":"ContainerDied","Data":"f6d0dbd4d9221789d4645a3432b03191aa45b6bd2365d678cd7959ccd7dd2c65"}
Oct 30 19:15:19.368178 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.368159    4924 scope.go:122] "RemoveContainer" containerID="f6d0dbd4d9221789d4645a3432b03191aa45b6bd2365d678cd7959ccd7dd2c65"
Oct 30 19:15:19.368521 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:19.368498    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx_odh-ai-gateway-infra(70bd8cf2-4a28-46e2-8223-27e0890f7444)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx" podUID="70bd8cf2-4a28-46e2-8223-27e0890f7444"
Oct 30 19:15:19.371130 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.371112    4924 generic.go:356] "Generic (PLEG): container finished" podID="534a63d0-c4b9-48cb-88f0-bed24a366017" containerID="47ef9a29fc3f3767a342eab9064f71a2693591511a4e8ffcfb8f7437beaf5c38" exitCode=60
Oct 30 19:15:19.371218 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.371191    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g" event={"ID":"534a63d0-c4b9-48cb-88f0-bed24a366017","Type":"ContainerDied","Data":"47ef9a29fc3f3767a342eab9064f71a2693591511a4e8ffcfb8f7437beaf5c38"}
Oct 30 19:15:19.371576 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.371556    4924 scope.go:122] "RemoveContainer" containerID="47ef9a29fc3f3767a342eab9064f71a2693591511a4e8ffcfb8f7437beaf5c38"
Oct 30 19:15:19.371861 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:19.371841    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g_odh-ai-gateway-infra(534a63d0-c4b9-48cb-88f0-bed24a366017)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g" podUID="534a63d0-c4b9-48cb-88f0-bed24a366017"
Oct 30 19:15:19.373078 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.373046    4924 generic.go:356] "Generic (PLEG): container finished" podID="e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1" containerID="33709958feeb25469ed22146144fc0df59a89bc2f73feaac444e72464b3e02e5" exitCode=60
Oct 30 19:15:19.373168 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.373090    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr" event={"ID":"e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1","Type":"ContainerDied","Data":"33709958feeb25469ed22146144fc0df59a89bc2f73feaac444e72464b3e02e5"}
Oct 30 19:15:19.373447 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.373429    4924 scope.go:122] "RemoveContainer" containerID="33709958feeb25469ed22146144fc0df59a89bc2f73feaac444e72464b3e02e5"
Oct 30 19:15:19.373759 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:19.373738    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr_odh-ai-gateway-infra(e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr" podUID="e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1"
Oct 30 19:15:19.377229 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.377203    4924 scope.go:122] "RemoveContainer" containerID="7cc084c75a58c0dd5d877b9000b5d1f08f4f3d6055053ea3391c84f0cb64c654"
Oct 30 19:15:19.386869 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.386838    4924 scope.go:122] "RemoveContainer" containerID="f6992130696f4be8093f28f8409e9eac848b44c35738d787c01c332a251c9c70"
Oct 30 19:15:19.402706 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.402683    4924 scope.go:122] "RemoveContainer" containerID="689804f1c62420b6997dd80a71df8531bf7e6d9c8804f87f7f689ad0d93834b7"
Oct 30 19:15:19.417690 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.417672    4924 scope.go:122] "RemoveContainer" containerID="e880e44d5bd8cf65996036201caa9aff1e994be589d9fcb42526397c3c9d72a5"
Oct 30 19:15:19.466305 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.466214    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx"]
Oct 30 19:15:19.515031 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.515005    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg"
Oct 30 19:15:19.600463 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.600417    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9d33ee4-cc7c-4043-a179-65094f01ca5f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d33ee4-cc7c-4043-a179-65094f01ca5f-maas-api-service-ca\") pod \"d9d33ee4-cc7c-4043-a179-65094f01ca5f\" (UID: \"d9d33ee4-cc7c-4043-a179-65094f01ca5f\") "
Oct 30 19:15:19.600634 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.600558    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9d33ee4-cc7c-4043-a179-65094f01ca5f-kube-api-access-8gvq2\" (UniqueName: \"kubernetes.io/projected/d9d33ee4-cc7c-4043-a179-65094f01ca5f-kube-api-access-8gvq2\") pod \"d9d33ee4-cc7c-4043-a179-65094f01ca5f\" (UID: \"d9d33ee4-cc7c-4043-a179-65094f01ca5f\") "
Oct 30 19:15:19.600748 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.600725    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9d33ee4-cc7c-4043-a179-65094f01ca5f-maas-api-service-ca" pod "d9d33ee4-cc7c-4043-a179-65094f01ca5f" (UID: "d9d33ee4-cc7c-4043-a179-65094f01ca5f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:19.601039 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.601025    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d33ee4-cc7c-4043-a179-65094f01ca5f-maas-api-service-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:19.602853 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.602828    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d33ee4-cc7c-4043-a179-65094f01ca5f-kube-api-access-8gvq2" pod "d9d33ee4-cc7c-4043-a179-65094f01ca5f" (UID: "d9d33ee4-cc7c-4043-a179-65094f01ca5f"). InnerVolumeSpecName "kube-api-access-8gvq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:19.702361 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.702328    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gvq2\" (UniqueName: \"kubernetes.io/projected/d9d33ee4-cc7c-4043-a179-65094f01ca5f-kube-api-access-8gvq2\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:19.800266 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.800235    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:19.892914 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:19.892882    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g"]
Oct 30 19:15:20.045556 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.045472    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96"]
Oct 30 19:15:20.048982 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.048959    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f75d5f-29889795-r8j96"]
Oct 30 19:15:20.481118 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.474708    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg"
Oct 30 19:15:20.481118 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.474718    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg" event={"ID":"d9d33ee4-cc7c-4043-a179-65094f01ca5f","Type":"ContainerDied","Data":"1921d8044b03cc57726adbaba87e3bb8789c204f1ed47b691827a1eb4da721cc"}
Oct 30 19:15:20.481118 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.474797    4924 scope.go:122] "RemoveContainer" containerID="4dceb8745a73ad7bd37294ff7c0b41131be1d0498ef5c741a8d869a688f0e5e5"
Oct 30 19:15:20.489987 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.488532    4924 generic.go:356] "Generic (PLEG): container finished" podID="142bd310-9137-4db5-a5e5-27fcf478fbb5" containerID="1afc3451bd896234f515a099df9a75b0fbf1002797d835261775612352235d1a" exitCode=60
Oct 30 19:15:20.489987 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.488577    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q" event={"ID":"142bd310-9137-4db5-a5e5-27fcf478fbb5","Type":"ContainerDied","Data":"1afc3451bd896234f515a099df9a75b0fbf1002797d835261775612352235d1a"}
Oct 30 19:15:20.489987 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.489141    4924 scope.go:122] "RemoveContainer" containerID="1afc3451bd896234f515a099df9a75b0fbf1002797d835261775612352235d1a"
Oct 30 19:15:20.489987 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:20.489621    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889795-bjw2q_odh-ai-gateway-infra(142bd310-9137-4db5-a5e5-27fcf478fbb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q" podUID="142bd310-9137-4db5-a5e5-27fcf478fbb5"
Oct 30 19:15:20.493228 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.493199    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q"]
Oct 30 19:15:20.512997 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.512936    4924 scope.go:122] "RemoveContainer" containerID="8abffc49dc96cbf26716d2ae6d38998cdd29acb4dd1fd8ad423403b31b579b6d"
Oct 30 19:15:20.692950 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.692920    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66ebbd68-lrwl9"]
Oct 30 19:15:20.693947 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.693923    4924 cpu_manager.go:414] "containerMap: removing container" podUID="7f9d64eb-f439-47cc-8f2c-d7eaf03fd573" containerName="cleanup"
Oct 30 19:15:20.694039 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.693949    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f9d64eb-f439-47cc-8f2c-d7eaf03fd573" containerName="cleanup"
Oct 30 19:15:20.694039 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.693987    4924 cpu_manager.go:414] "containerMap: removing container" podUID="d9d33ee4-cc7c-4043-a179-65094f01ca5f" containerName="cleanup"
Oct 30 19:15:20.694039 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.693995    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9d33ee4-cc7c-4043-a179-65094f01ca5f" containerName="cleanup"
Oct 30 19:15:20.694039 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694009    4924 cpu_manager.go:414] "containerMap: removing container" podUID="000b4c54-e96e-497e-b489-504f2f52094c" containerName="cleanup"
Oct 30 19:15:20.694039 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694016    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="000b4c54-e96e-497e-b489-504f2f52094c" containerName="cleanup"
Oct 30 19:15:20.694039 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694032    4924 cpu_manager.go:414] "containerMap: removing container" podUID="d9d33ee4-cc7c-4043-a179-65094f01ca5f" containerName="cleanup"
Oct 30 19:15:20.694039 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694039    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9d33ee4-cc7c-4043-a179-65094f01ca5f" containerName="cleanup"
Oct 30 19:15:20.694294 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694081    4924 cpu_manager.go:414] "containerMap: removing container" podUID="4ea62e28-48cf-40e4-9de4-1b767bd839ce" containerName="cleanup"
Oct 30 19:15:20.694294 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694089    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea62e28-48cf-40e4-9de4-1b767bd839ce" containerName="cleanup"
Oct 30 19:15:20.694294 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694113    4924 cpu_manager.go:414] "containerMap: removing container" podUID="51b9a4f5-d2e7-462e-ab62-03ef1a125d89" containerName="revoke-keys"
Oct 30 19:15:20.694294 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694121    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51b9a4f5-d2e7-462e-ab62-03ef1a125d89" containerName="revoke-keys"
Oct 30 19:15:20.694294 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694132    4924 cpu_manager.go:414] "containerMap: removing container" podUID="d9d33ee4-cc7c-4043-a179-65094f01ca5f" containerName="cleanup"
Oct 30 19:15:20.694294 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694139    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9d33ee4-cc7c-4043-a179-65094f01ca5f" containerName="cleanup"
Oct 30 19:15:20.694294 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694152    4924 cpu_manager.go:414] "containerMap: removing container" podUID="7f9d64eb-f439-47cc-8f2c-d7eaf03fd573" containerName="cleanup"
Oct 30 19:15:20.694294 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694159    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f9d64eb-f439-47cc-8f2c-d7eaf03fd573" containerName="cleanup"
Oct 30 19:15:20.694294 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694175    4924 cpu_manager.go:414] "containerMap: removing container" podUID="0b676034-384c-4c43-a8f5-991977f109ce" containerName="payload-processing"
Oct 30 19:15:20.694294 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694182    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b676034-384c-4c43-a8f5-991977f109ce" containerName="payload-processing"
Oct 30 19:15:20.694294 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694216    4924 cpu_manager.go:414] "containerMap: removing container" podUID="4ea62e28-48cf-40e4-9de4-1b767bd839ce" containerName="cleanup"
Oct 30 19:15:20.694294 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694224    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea62e28-48cf-40e4-9de4-1b767bd839ce" containerName="cleanup"
Oct 30 19:15:20.694651 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694408    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea62e28-48cf-40e4-9de4-1b767bd839ce" containerName="cleanup"
Oct 30 19:15:20.694651 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694423    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f9d64eb-f439-47cc-8f2c-d7eaf03fd573" containerName="cleanup"
Oct 30 19:15:20.694651 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694433    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="51b9a4f5-d2e7-462e-ab62-03ef1a125d89" containerName="revoke-keys"
Oct 30 19:15:20.694651 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694445    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9d33ee4-cc7c-4043-a179-65094f01ca5f" containerName="cleanup"
Oct 30 19:15:20.694651 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694452    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f9d64eb-f439-47cc-8f2c-d7eaf03fd573" containerName="cleanup"
Oct 30 19:15:20.694651 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694461    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b676034-384c-4c43-a8f5-991977f109ce" containerName="payload-processing"
Oct 30 19:15:20.694651 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694472    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="000b4c54-e96e-497e-b489-504f2f52094c" containerName="cleanup"
Oct 30 19:15:20.694651 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694484    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea62e28-48cf-40e4-9de4-1b767bd839ce" containerName="cleanup"
Oct 30 19:15:20.694651 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694494    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f9d64eb-f439-47cc-8f2c-d7eaf03fd573" containerName="cleanup"
Oct 30 19:15:20.694651 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694509    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9d33ee4-cc7c-4043-a179-65094f01ca5f" containerName="cleanup"
Oct 30 19:15:20.694651 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.694520    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9d33ee4-cc7c-4043-a179-65094f01ca5f" containerName="cleanup"
Oct 30 19:15:20.700188 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.700165    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ebbd68-lrwl9"
Oct 30 19:15:20.721492 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.721464    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:20.729052 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.729007    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx"
Oct 30 19:15:20.735267 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.735210    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr"]
Oct 30 19:15:20.756495 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.754495    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:15:20.768277 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.765503    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6d86012-cbvvz"]
Oct 30 19:15:20.768277 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.766553    4924 cpu_manager.go:414] "containerMap: removing container" podUID="7f9d64eb-f439-47cc-8f2c-d7eaf03fd573" containerName="cleanup"
Oct 30 19:15:20.768277 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.766575    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f9d64eb-f439-47cc-8f2c-d7eaf03fd573" containerName="cleanup"
Oct 30 19:15:20.768277 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.766632    4924 cpu_manager.go:414] "containerMap: removing container" podUID="70bd8cf2-4a28-46e2-8223-27e0890f7444" containerName="cleanup"
Oct 30 19:15:20.768277 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.766640    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70bd8cf2-4a28-46e2-8223-27e0890f7444" containerName="cleanup"
Oct 30 19:15:20.768277 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.766683    4924 cpu_manager.go:414] "containerMap: removing container" podUID="70bd8cf2-4a28-46e2-8223-27e0890f7444" containerName="cleanup"
Oct 30 19:15:20.768277 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.766691    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70bd8cf2-4a28-46e2-8223-27e0890f7444" containerName="cleanup"
Oct 30 19:15:20.768277 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.766712    4924 cpu_manager.go:414] "containerMap: removing container" podUID="70bd8cf2-4a28-46e2-8223-27e0890f7444" containerName="cleanup"
Oct 30 19:15:20.768277 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.766719    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70bd8cf2-4a28-46e2-8223-27e0890f7444" containerName="cleanup"
Oct 30 19:15:20.768277 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.766734    4924 cpu_manager.go:414] "containerMap: removing container" podUID="4ea62e28-48cf-40e4-9de4-1b767bd839ce" containerName="cleanup"
Oct 30 19:15:20.768277 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.766741    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea62e28-48cf-40e4-9de4-1b767bd839ce" containerName="cleanup"
Oct 30 19:15:20.768277 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.766893    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="70bd8cf2-4a28-46e2-8223-27e0890f7444" containerName="cleanup"
Oct 30 19:15:20.768277 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.766904    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="70bd8cf2-4a28-46e2-8223-27e0890f7444" containerName="cleanup"
Oct 30 19:15:20.768277 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.766913    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea62e28-48cf-40e4-9de4-1b767bd839ce" containerName="cleanup"
Oct 30 19:15:20.768277 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.766932    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="70bd8cf2-4a28-46e2-8223-27e0890f7444" containerName="cleanup"
Oct 30 19:15:20.771291 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.771262    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66ebbd68-lrwl9"]
Oct 30 19:15:20.772481 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.771379    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d86012-cbvvz"
Oct 30 19:15:20.783044 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.781158    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70bd8cf2-4a28-46e2-8223-27e0890f7444-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70bd8cf2-4a28-46e2-8223-27e0890f7444-maas-api-service-ca\") pod \"70bd8cf2-4a28-46e2-8223-27e0890f7444\" (UID: \"70bd8cf2-4a28-46e2-8223-27e0890f7444\") "
Oct 30 19:15:20.783044 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.781364    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70bd8cf2-4a28-46e2-8223-27e0890f7444-kube-api-access-t6c5h\" (UniqueName: \"kubernetes.io/projected/70bd8cf2-4a28-46e2-8223-27e0890f7444-kube-api-access-t6c5h\") pod \"70bd8cf2-4a28-46e2-8223-27e0890f7444\" (UID: \"70bd8cf2-4a28-46e2-8223-27e0890f7444\") "
Oct 30 19:15:20.783044 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.781661    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4087501-1956-4f6e-8b87-9223565cce59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66ebbd68-lrwl9\" (UID: \"a4087501-1956-4f6e-8b87-9223565cce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ebbd68-lrwl9"
Oct 30 19:15:20.783044 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.781838    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k44c4\" (UniqueName: \"kubernetes.io/projected/a4087501-1956-4f6e-8b87-9223565cce59-kube-api-access-k44c4\") pod \"maas-api-revoke-sub-66ebbd68-lrwl9\" (UID: \"a4087501-1956-4f6e-8b87-9223565cce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ebbd68-lrwl9"
Oct 30 19:15:20.783044 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.781889    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k49ws\" (UniqueName: \"kubernetes.io/projected/c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06-kube-api-access-k49ws\") pod \"maas-api-revoke-sub-a6d86012-cbvvz\" (UID: \"c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d86012-cbvvz"
Oct 30 19:15:20.783044 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.781981    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4087501-1956-4f6e-8b87-9223565cce59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66ebbd68-lrwl9\" (UID: \"a4087501-1956-4f6e-8b87-9223565cce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ebbd68-lrwl9"
Oct 30 19:15:20.783044 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.782053    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6d86012-cbvvz\" (UID: \"c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d86012-cbvvz"
Oct 30 19:15:20.783044 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.782175    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6d86012-cbvvz\" (UID: \"c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d86012-cbvvz"
Oct 30 19:15:20.784152 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.783818    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70bd8cf2-4a28-46e2-8223-27e0890f7444-maas-api-service-ca" pod "70bd8cf2-4a28-46e2-8223-27e0890f7444" (UID: "70bd8cf2-4a28-46e2-8223-27e0890f7444"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:20.786552 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.786510    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70bd8cf2-4a28-46e2-8223-27e0890f7444-kube-api-access-t6c5h" pod "70bd8cf2-4a28-46e2-8223-27e0890f7444" (UID: "70bd8cf2-4a28-46e2-8223-27e0890f7444"). InnerVolumeSpecName "kube-api-access-t6c5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:20.791695 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.791617    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g"
Oct 30 19:15:20.815171 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.815143    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6d86012-cbvvz"]
Oct 30 19:15:20.846383 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.846354    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ea62e28-48cf-40e4-9de4-1b767bd839ce" path="/var/lib/kubelet/pods/4ea62e28-48cf-40e4-9de4-1b767bd839ce/volumes"
Oct 30 19:15:20.849383 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.849363    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh"]
Oct 30 19:15:20.853732 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.853705    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg"]
Oct 30 19:15:20.866570 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.866545    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec8dab-29889795-jbxjg"]
Oct 30 19:15:20.882911 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.882884    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/534a63d0-c4b9-48cb-88f0-bed24a366017-kube-api-access-5hj8r\" (UniqueName: \"kubernetes.io/projected/534a63d0-c4b9-48cb-88f0-bed24a366017-kube-api-access-5hj8r\") pod \"534a63d0-c4b9-48cb-88f0-bed24a366017\" (UID: \"534a63d0-c4b9-48cb-88f0-bed24a366017\") "
Oct 30 19:15:20.883059 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.883017    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/534a63d0-c4b9-48cb-88f0-bed24a366017-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/534a63d0-c4b9-48cb-88f0-bed24a366017-maas-api-service-ca\") pod \"534a63d0-c4b9-48cb-88f0-bed24a366017\" (UID: \"534a63d0-c4b9-48cb-88f0-bed24a366017\") "
Oct 30 19:15:20.883059 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.883298    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4087501-1956-4f6e-8b87-9223565cce59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66ebbd68-lrwl9\" (UID: \"a4087501-1956-4f6e-8b87-9223565cce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ebbd68-lrwl9"
Oct 30 19:15:20.883059 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.883362    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6d86012-cbvvz\" (UID: \"c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d86012-cbvvz"
Oct 30 19:15:20.883623 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.883438    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6d86012-cbvvz\" (UID: \"c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d86012-cbvvz"
Oct 30 19:15:20.883623 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.883525    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4087501-1956-4f6e-8b87-9223565cce59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66ebbd68-lrwl9\" (UID: \"a4087501-1956-4f6e-8b87-9223565cce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ebbd68-lrwl9"
Oct 30 19:15:20.883742 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.883695    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k44c4\" (UniqueName: \"kubernetes.io/projected/a4087501-1956-4f6e-8b87-9223565cce59-kube-api-access-k44c4\") pod \"maas-api-revoke-sub-66ebbd68-lrwl9\" (UID: \"a4087501-1956-4f6e-8b87-9223565cce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ebbd68-lrwl9"
Oct 30 19:15:20.883742 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.883732    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k49ws\" (UniqueName: \"kubernetes.io/projected/c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06-kube-api-access-k49ws\") pod \"maas-api-revoke-sub-a6d86012-cbvvz\" (UID: \"c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d86012-cbvvz"
Oct 30 19:15:20.883848 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.883836    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70bd8cf2-4a28-46e2-8223-27e0890f7444-maas-api-service-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.883902 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.883853    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6c5h\" (UniqueName: \"kubernetes.io/projected/70bd8cf2-4a28-46e2-8223-27e0890f7444-kube-api-access-t6c5h\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.884813 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.884351    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4087501-1956-4f6e-8b87-9223565cce59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66ebbd68-lrwl9\" (UID: \"a4087501-1956-4f6e-8b87-9223565cce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ebbd68-lrwl9"
Oct 30 19:15:20.884813 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.884767    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6d86012-cbvvz\" (UID: \"c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d86012-cbvvz"
Oct 30 19:15:20.887326 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.887284    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/534a63d0-c4b9-48cb-88f0-bed24a366017-kube-api-access-5hj8r" pod "534a63d0-c4b9-48cb-88f0-bed24a366017" (UID: "534a63d0-c4b9-48cb-88f0-bed24a366017"). InnerVolumeSpecName "kube-api-access-5hj8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:20.889479 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.889435    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/534a63d0-c4b9-48cb-88f0-bed24a366017-maas-api-service-ca" pod "534a63d0-c4b9-48cb-88f0-bed24a366017" (UID: "534a63d0-c4b9-48cb-88f0-bed24a366017"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:20.892957 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.892914    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4087501-1956-4f6e-8b87-9223565cce59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66ebbd68-lrwl9\" (UID: \"a4087501-1956-4f6e-8b87-9223565cce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ebbd68-lrwl9"
Oct 30 19:15:20.909233 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.909176    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:20.916438 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.916409    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6d86012-cbvvz\" (UID: \"c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d86012-cbvvz"
Oct 30 19:15:20.950105 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.950028    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k44c4\" (UniqueName: \"kubernetes.io/projected/a4087501-1956-4f6e-8b87-9223565cce59-kube-api-access-k44c4\") pod \"maas-api-revoke-sub-66ebbd68-lrwl9\" (UID: \"a4087501-1956-4f6e-8b87-9223565cce59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ebbd68-lrwl9"
Oct 30 19:15:20.951271 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.951243    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k49ws\" (UniqueName: \"kubernetes.io/projected/c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06-kube-api-access-k49ws\") pod \"maas-api-revoke-sub-a6d86012-cbvvz\" (UID: \"c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d86012-cbvvz"
Oct 30 19:15:20.971693 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.971657    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr"
Oct 30 19:15:20.985081 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.984339    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1-kube-api-access-n6n9h\" (UniqueName: \"kubernetes.io/projected/e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1-kube-api-access-n6n9h\") pod \"e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1\" (UID: \"e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1\") "
Oct 30 19:15:20.985081 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.984557    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1-maas-api-service-ca\") pod \"e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1\" (UID: \"e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1\") "
Oct 30 19:15:20.985307 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.985204    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/534a63d0-c4b9-48cb-88f0-bed24a366017-maas-api-service-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.985307 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.985224    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hj8r\" (UniqueName: \"kubernetes.io/projected/534a63d0-c4b9-48cb-88f0-bed24a366017-kube-api-access-5hj8r\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:20.985307 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.985257    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1-maas-api-service-ca" pod "e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1" (UID: "e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:20.996240 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:20.992267    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1-kube-api-access-n6n9h" pod "e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1" (UID: "e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1"). InnerVolumeSpecName "kube-api-access-n6n9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:21.024289 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.024250    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh"
Oct 30 19:15:21.086308 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.086272    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9aa4147-99fd-4ee4-9dd0-43ae21407002-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9aa4147-99fd-4ee4-9dd0-43ae21407002-maas-api-service-ca\") pod \"d9aa4147-99fd-4ee4-9dd0-43ae21407002\" (UID: \"d9aa4147-99fd-4ee4-9dd0-43ae21407002\") "
Oct 30 19:15:21.086477 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.086344    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9aa4147-99fd-4ee4-9dd0-43ae21407002-kube-api-access-bhnl5\" (UniqueName: \"kubernetes.io/projected/d9aa4147-99fd-4ee4-9dd0-43ae21407002-kube-api-access-bhnl5\") pod \"d9aa4147-99fd-4ee4-9dd0-43ae21407002\" (UID: \"d9aa4147-99fd-4ee4-9dd0-43ae21407002\") "
Oct 30 19:15:21.086816 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.086786    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9aa4147-99fd-4ee4-9dd0-43ae21407002-maas-api-service-ca" pod "d9aa4147-99fd-4ee4-9dd0-43ae21407002" (UID: "d9aa4147-99fd-4ee4-9dd0-43ae21407002"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:21.087123 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.087080    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9aa4147-99fd-4ee4-9dd0-43ae21407002-maas-api-service-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.087123 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.087110    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6n9h\" (UniqueName: \"kubernetes.io/projected/e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1-kube-api-access-n6n9h\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.087279 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.087126    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1-maas-api-service-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.090046 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.090019    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9aa4147-99fd-4ee4-9dd0-43ae21407002-kube-api-access-bhnl5" pod "d9aa4147-99fd-4ee4-9dd0-43ae21407002" (UID: "d9aa4147-99fd-4ee4-9dd0-43ae21407002"). InnerVolumeSpecName "kube-api-access-bhnl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:21.120230 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.120205    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ebbd68-lrwl9"
Oct 30 19:15:21.127278 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.127248    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:21.130574 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.130550    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d86012-cbvvz"
Oct 30 19:15:21.194105 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.191198    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhnl5\" (UniqueName: \"kubernetes.io/projected/d9aa4147-99fd-4ee4-9dd0-43ae21407002-kube-api-access-bhnl5\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.498657 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.498579    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx"
Oct 30 19:15:21.498657 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.498609    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx" event={"ID":"70bd8cf2-4a28-46e2-8223-27e0890f7444","Type":"ContainerDied","Data":"687a88ebc5ccf839cb6f9ec91ceb5a1d248e463023ef8ac6f2822c2c41a94ce0"}
Oct 30 19:15:21.499228 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.498667    4924 scope.go:122] "RemoveContainer" containerID="f6d0dbd4d9221789d4645a3432b03191aa45b6bd2365d678cd7959ccd7dd2c65"
Oct 30 19:15:21.500204 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.500184    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g"
Oct 30 19:15:21.500308 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.500221    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g" event={"ID":"534a63d0-c4b9-48cb-88f0-bed24a366017","Type":"ContainerDied","Data":"1aac7f0d0252396cb0f37011ca326a2cd7a7edbe59c76ea8cff2a61653b8416e"}
Oct 30 19:15:21.502278 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.502256    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr"
Oct 30 19:15:21.502389 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.502272    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr" event={"ID":"e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1","Type":"ContainerDied","Data":"ed90f71ec3bc6cd73ff014b6d4cc4c1b99cfe3a528c01954328dcaa5e8d5351b"}
Oct 30 19:15:21.511441 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.511424    4924 scope.go:122] "RemoveContainer" containerID="47ef9a29fc3f3767a342eab9064f71a2693591511a4e8ffcfb8f7437beaf5c38"
Oct 30 19:15:21.512121 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.512097    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh" event={"ID":"d9aa4147-99fd-4ee4-9dd0-43ae21407002","Type":"ContainerDied","Data":"9fce9c60dd9ef9f91048cb94033b83466e490429d0e8863b31e5ad7bc38b752f"}
Oct 30 19:15:21.512205 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.512198    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh"
Oct 30 19:15:21.525029 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.525000    4924 scope.go:122] "RemoveContainer" containerID="33709958feeb25469ed22146144fc0df59a89bc2f73feaac444e72464b3e02e5"
Oct 30 19:15:21.534474 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.534452    4924 scope.go:122] "RemoveContainer" containerID="ec53e51287de5c802198e31d39277109202a4f9142b30f8eaa838ef369266049"
Oct 30 19:15:21.608133 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:15:21.606326    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4087501_1956_4f6e_8b87_9223565cce59.slice/crio-6dc449f7c1d18cc9760065676c34c4ae1f2cfa457df4f40e78045da2754e66f6 WatchSource:0}: Error finding container 6dc449f7c1d18cc9760065676c34c4ae1f2cfa457df4f40e78045da2754e66f6: Status 404 returned error can't find the container with id 6dc449f7c1d18cc9760065676c34c4ae1f2cfa457df4f40e78045da2754e66f6
Oct 30 19:15:21.640803 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.640770    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:21.649307 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.649143    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66ebbd68-lrwl9"]
Oct 30 19:15:21.656115 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.653171    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6d86012-cbvvz"]
Oct 30 19:15:21.691368 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.691347    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq"]
Oct 30 19:15:21.692012 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.691993    4924 cpu_manager.go:414] "containerMap: removing container" podUID="534a63d0-c4b9-48cb-88f0-bed24a366017" containerName="cleanup"
Oct 30 19:15:21.692124 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.692015    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="534a63d0-c4b9-48cb-88f0-bed24a366017" containerName="cleanup"
Oct 30 19:15:21.692124 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.692038    4924 cpu_manager.go:414] "containerMap: removing container" podUID="d9aa4147-99fd-4ee4-9dd0-43ae21407002" containerName="cleanup"
Oct 30 19:15:21.692124 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.692047    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9aa4147-99fd-4ee4-9dd0-43ae21407002" containerName="cleanup"
Oct 30 19:15:21.692124 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.692079    4924 cpu_manager.go:414] "containerMap: removing container" podUID="e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1" containerName="cleanup"
Oct 30 19:15:21.692124 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.692089    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1" containerName="cleanup"
Oct 30 19:15:21.692124 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.692110    4924 cpu_manager.go:414] "containerMap: removing container" podUID="e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1" containerName="cleanup"
Oct 30 19:15:21.692124 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.692118    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1" containerName="cleanup"
Oct 30 19:15:21.692486 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.692180    4924 cpu_manager.go:414] "containerMap: removing container" podUID="534a63d0-c4b9-48cb-88f0-bed24a366017" containerName="cleanup"
Oct 30 19:15:21.692486 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.692188    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="534a63d0-c4b9-48cb-88f0-bed24a366017" containerName="cleanup"
Oct 30 19:15:21.692486 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.692203    4924 cpu_manager.go:414] "containerMap: removing container" podUID="d9aa4147-99fd-4ee4-9dd0-43ae21407002" containerName="cleanup"
Oct 30 19:15:21.692486 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.692211    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9aa4147-99fd-4ee4-9dd0-43ae21407002" containerName="cleanup"
Oct 30 19:15:21.692486 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.692372    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9aa4147-99fd-4ee4-9dd0-43ae21407002" containerName="cleanup"
Oct 30 19:15:21.692486 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.692385    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="534a63d0-c4b9-48cb-88f0-bed24a366017" containerName="cleanup"
Oct 30 19:15:21.692486 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.692397    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9aa4147-99fd-4ee4-9dd0-43ae21407002" containerName="cleanup"
Oct 30 19:15:21.692486 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.692408    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="534a63d0-c4b9-48cb-88f0-bed24a366017" containerName="cleanup"
Oct 30 19:15:21.692486 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.692419    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9aa4147-99fd-4ee4-9dd0-43ae21407002" containerName="cleanup"
Oct 30 19:15:21.692486 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.692429    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1" containerName="cleanup"
Oct 30 19:15:21.692486 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.692439    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1" containerName="cleanup"
Oct 30 19:15:21.702183 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.702162    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq"
Oct 30 19:15:21.756090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.754504    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq"]
Oct 30 19:15:21.782183 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.782149    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q"
Oct 30 19:15:21.788433 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.788402    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx"]
Oct 30 19:15:21.798868 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.798844    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/142bd310-9137-4db5-a5e5-27fcf478fbb5-kube-api-access-fbhl8\" (UniqueName: \"kubernetes.io/projected/142bd310-9137-4db5-a5e5-27fcf478fbb5-kube-api-access-fbhl8\") pod \"142bd310-9137-4db5-a5e5-27fcf478fbb5\" (UID: \"142bd310-9137-4db5-a5e5-27fcf478fbb5\") "
Oct 30 19:15:21.798996 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.798983    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/142bd310-9137-4db5-a5e5-27fcf478fbb5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/142bd310-9137-4db5-a5e5-27fcf478fbb5-maas-api-service-ca\") pod \"142bd310-9137-4db5-a5e5-27fcf478fbb5\" (UID: \"142bd310-9137-4db5-a5e5-27fcf478fbb5\") "
Oct 30 19:15:21.799252 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.799232    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58181eb5-380f-49a1-bd72-b543779e9410-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq\" (UID: \"58181eb5-380f-49a1-bd72-b543779e9410\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq"
Oct 30 19:15:21.799380 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.799315    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58181eb5-380f-49a1-bd72-b543779e9410-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq\" (UID: \"58181eb5-380f-49a1-bd72-b543779e9410\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq"
Oct 30 19:15:21.799454 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.799408    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/142bd310-9137-4db5-a5e5-27fcf478fbb5-maas-api-service-ca" pod "142bd310-9137-4db5-a5e5-27fcf478fbb5" (UID: "142bd310-9137-4db5-a5e5-27fcf478fbb5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:21.799518 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.799468    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn757\" (UniqueName: \"kubernetes.io/projected/58181eb5-380f-49a1-bd72-b543779e9410-kube-api-access-tn757\") pod \"maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq\" (UID: \"58181eb5-380f-49a1-bd72-b543779e9410\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq"
Oct 30 19:15:21.799620 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.799605    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/142bd310-9137-4db5-a5e5-27fcf478fbb5-maas-api-service-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.801626 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.801599    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/142bd310-9137-4db5-a5e5-27fcf478fbb5-kube-api-access-fbhl8" pod "142bd310-9137-4db5-a5e5-27fcf478fbb5" (UID: "142bd310-9137-4db5-a5e5-27fcf478fbb5"). InnerVolumeSpecName "kube-api-access-fbhl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:21.818902 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.818873    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f5705f-29889795-6lgfx"]
Oct 30 19:15:21.900929 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.900896    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58181eb5-380f-49a1-bd72-b543779e9410-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq\" (UID: \"58181eb5-380f-49a1-bd72-b543779e9410\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq"
Oct 30 19:15:21.901147 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.901002    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn757\" (UniqueName: \"kubernetes.io/projected/58181eb5-380f-49a1-bd72-b543779e9410-kube-api-access-tn757\") pod \"maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq\" (UID: \"58181eb5-380f-49a1-bd72-b543779e9410\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq"
Oct 30 19:15:21.901270 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.901248    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58181eb5-380f-49a1-bd72-b543779e9410-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq\" (UID: \"58181eb5-380f-49a1-bd72-b543779e9410\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq"
Oct 30 19:15:21.901416 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.901399    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbhl8\" (UniqueName: \"kubernetes.io/projected/142bd310-9137-4db5-a5e5-27fcf478fbb5-kube-api-access-fbhl8\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:21.901745 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.901728    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58181eb5-380f-49a1-bd72-b543779e9410-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq\" (UID: \"58181eb5-380f-49a1-bd72-b543779e9410\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq"
Oct 30 19:15:21.903849 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.903821    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58181eb5-380f-49a1-bd72-b543779e9410-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq\" (UID: \"58181eb5-380f-49a1-bd72-b543779e9410\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq"
Oct 30 19:15:21.955651 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:21.955628    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn757\" (UniqueName: \"kubernetes.io/projected/58181eb5-380f-49a1-bd72-b543779e9410-kube-api-access-tn757\") pod \"maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq\" (UID: \"58181eb5-380f-49a1-bd72-b543779e9410\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq"
Oct 30 19:15:22.038808 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:22.038734    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g"]
Oct 30 19:15:22.050616 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:22.050592    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-568107-29889795-ppr4g"]
Oct 30 19:15:22.095414 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:22.095389    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq"
Oct 30 19:15:22.214716 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:22.214687    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr"]
Oct 30 19:15:22.222843 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:22.222817    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-314bd6fa-29889795-qqtvr"]
Oct 30 19:15:22.387165 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:22.387136    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh"]
Oct 30 19:15:22.393103 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:22.393060    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a1617e-29889795-svwmh"]
Oct 30 19:15:22.468858 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:22.468829    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq"]
Oct 30 19:15:22.469387 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:15:22.469361    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58181eb5_380f_49a1_bd72_b543779e9410.slice/crio-bc1fc271727c5aa30555415fe9cae0ea712c60ea2b03a1bf6972ffea297f63e3 WatchSource:0}: Error finding container bc1fc271727c5aa30555415fe9cae0ea712c60ea2b03a1bf6972ffea297f63e3: Status 404 returned error can't find the container with id bc1fc271727c5aa30555415fe9cae0ea712c60ea2b03a1bf6972ffea297f63e3
Oct 30 19:15:22.527481 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:22.527450    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q"
Oct 30 19:15:22.527481 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:22.527465    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q" event={"ID":"142bd310-9137-4db5-a5e5-27fcf478fbb5","Type":"ContainerDied","Data":"25996c8034d5d4111f363ba853346ddd2a205b45a87004f165ad7648cf3eefd7"}
Oct 30 19:15:22.527921 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:22.527526    4924 scope.go:122] "RemoveContainer" containerID="1afc3451bd896234f515a099df9a75b0fbf1002797d835261775612352235d1a"
Oct 30 19:15:22.530733 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:22.530705    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d86012-cbvvz" event={"ID":"c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06","Type":"ContainerStarted","Data":"ff16e300450d66861b648b25d698f026cc50115abfb9610a2be55ac0e71efae2"}
Oct 30 19:15:22.530922 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:22.530785    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d86012-cbvvz" event={"ID":"c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06","Type":"ContainerStarted","Data":"cf7b05214be31aca7edffe26b5d30d218cec03e6edee249e80e244bc1a677fa8"}
Oct 30 19:15:22.532588 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:22.532548    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ebbd68-lrwl9" event={"ID":"a4087501-1956-4f6e-8b87-9223565cce59","Type":"ContainerStarted","Data":"09cbd77013c968922d299888c5accb478bd453198ca4d16fda4be5dd5582bf19"}
Oct 30 19:15:22.532685 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:22.532605    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ebbd68-lrwl9" event={"ID":"a4087501-1956-4f6e-8b87-9223565cce59","Type":"ContainerStarted","Data":"6dc449f7c1d18cc9760065676c34c4ae1f2cfa457df4f40e78045da2754e66f6"}
Oct 30 19:15:22.534589 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:22.534553    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq" event={"ID":"58181eb5-380f-49a1-bd72-b543779e9410","Type":"ContainerStarted","Data":"bc1fc271727c5aa30555415fe9cae0ea712c60ea2b03a1bf6972ffea297f63e3"}
Oct 30 19:15:22.722882 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:22.722813    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d86012-cbvvz" podStartSLOduration=2.722795873 podStartE2EDuration="2.722795873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:22.722546281 +0000 UTC m=+2323.999596729" watchObservedRunningTime="2026-10-30 19:15:22.722795873 +0000 UTC m=+2323.999846316"
Oct 30 19:15:22.838430 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:22.838402    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="534a63d0-c4b9-48cb-88f0-bed24a366017" path="/var/lib/kubelet/pods/534a63d0-c4b9-48cb-88f0-bed24a366017/volumes"
Oct 30 19:15:22.838918 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:22.838904    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70bd8cf2-4a28-46e2-8223-27e0890f7444" path="/var/lib/kubelet/pods/70bd8cf2-4a28-46e2-8223-27e0890f7444/volumes"
Oct 30 19:15:22.839359 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:22.839345    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9aa4147-99fd-4ee4-9dd0-43ae21407002" path="/var/lib/kubelet/pods/d9aa4147-99fd-4ee4-9dd0-43ae21407002/volumes"
Oct 30 19:15:22.840180 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:22.840163    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9d33ee4-cc7c-4043-a179-65094f01ca5f" path="/var/lib/kubelet/pods/d9d33ee4-cc7c-4043-a179-65094f01ca5f/volumes"
Oct 30 19:15:22.840602 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:22.840588    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1" path="/var/lib/kubelet/pods/e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1/volumes"
Oct 30 19:15:22.841009 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:22.840993    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q"]
Oct 30 19:15:22.841093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:22.841023    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889795-bjw2q"]
Oct 30 19:15:22.941018 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:22.940930    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ebbd68-lrwl9" podStartSLOduration=2.940918131 podStartE2EDuration="2.940918131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:22.939563229 +0000 UTC m=+2324.216613674" watchObservedRunningTime="2026-10-30 19:15:22.940918131 +0000 UTC m=+2324.217968577"
Oct 30 19:15:23.545782 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:23.545749    4924 generic.go:356] "Generic (PLEG): container finished" podID="a4087501-1956-4f6e-8b87-9223565cce59" containerID="09cbd77013c968922d299888c5accb478bd453198ca4d16fda4be5dd5582bf19" exitCode=0
Oct 30 19:15:23.546210 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:23.545829    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ebbd68-lrwl9" event={"ID":"a4087501-1956-4f6e-8b87-9223565cce59","Type":"ContainerDied","Data":"09cbd77013c968922d299888c5accb478bd453198ca4d16fda4be5dd5582bf19"}
Oct 30 19:15:23.547479 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:23.547452    4924 generic.go:356] "Generic (PLEG): container finished" podID="58181eb5-380f-49a1-bd72-b543779e9410" containerID="8cec6bf9812cf618d47bd9ceba66988ff49d92d1c170ebcb0ffd9e71abd9081c" exitCode=0
Oct 30 19:15:23.547610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:23.547478    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq" event={"ID":"58181eb5-380f-49a1-bd72-b543779e9410","Type":"ContainerDied","Data":"8cec6bf9812cf618d47bd9ceba66988ff49d92d1c170ebcb0ffd9e71abd9081c"}
Oct 30 19:15:23.550920 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:23.550899    4924 generic.go:356] "Generic (PLEG): container finished" podID="c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06" containerID="ff16e300450d66861b648b25d698f026cc50115abfb9610a2be55ac0e71efae2" exitCode=0
Oct 30 19:15:23.551016 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:23.550934    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d86012-cbvvz" event={"ID":"c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06","Type":"ContainerDied","Data":"ff16e300450d66861b648b25d698f026cc50115abfb9610a2be55ac0e71efae2"}
Oct 30 19:15:24.777874 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.777844    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ebbd68-lrwl9"
Oct 30 19:15:24.813719 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.813695    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq"
Oct 30 19:15:24.817156 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.817136    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d86012-cbvvz"
Oct 30 19:15:24.835628 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.835580    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="142bd310-9137-4db5-a5e5-27fcf478fbb5" path="/var/lib/kubelet/pods/142bd310-9137-4db5-a5e5-27fcf478fbb5/volumes"
Oct 30 19:15:24.841351 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.841328    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06-maas-api-xks-ca\") pod \"c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06\" (UID: \"c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06\") "
Oct 30 19:15:24.841436 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.841364    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4087501-1956-4f6e-8b87-9223565cce59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4087501-1956-4f6e-8b87-9223565cce59-maas-api-xks-ca\") pod \"a4087501-1956-4f6e-8b87-9223565cce59\" (UID: \"a4087501-1956-4f6e-8b87-9223565cce59\") "
Oct 30 19:15:24.841436 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.841406    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4087501-1956-4f6e-8b87-9223565cce59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4087501-1956-4f6e-8b87-9223565cce59-maas-api-openshift-ca\") pod \"a4087501-1956-4f6e-8b87-9223565cce59\" (UID: \"a4087501-1956-4f6e-8b87-9223565cce59\") "
Oct 30 19:15:24.841565 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.841446    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06-kube-api-access-k49ws\" (UniqueName: \"kubernetes.io/projected/c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06-kube-api-access-k49ws\") pod \"c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06\" (UID: \"c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06\") "
Oct 30 19:15:24.841565 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.841507    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06-maas-api-openshift-ca\") pod \"c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06\" (UID: \"c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06\") "
Oct 30 19:15:24.841565 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.841547    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58181eb5-380f-49a1-bd72-b543779e9410-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58181eb5-380f-49a1-bd72-b543779e9410-maas-api-xks-ca\") pod \"58181eb5-380f-49a1-bd72-b543779e9410\" (UID: \"58181eb5-380f-49a1-bd72-b543779e9410\") "
Oct 30 19:15:24.841733 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.841685    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58181eb5-380f-49a1-bd72-b543779e9410-kube-api-access-tn757\" (UniqueName: \"kubernetes.io/projected/58181eb5-380f-49a1-bd72-b543779e9410-kube-api-access-tn757\") pod \"58181eb5-380f-49a1-bd72-b543779e9410\" (UID: \"58181eb5-380f-49a1-bd72-b543779e9410\") "
Oct 30 19:15:24.841832 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.841810    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4087501-1956-4f6e-8b87-9223565cce59-maas-api-openshift-ca" pod "a4087501-1956-4f6e-8b87-9223565cce59" (UID: "a4087501-1956-4f6e-8b87-9223565cce59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:24.841893 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.841848    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06-maas-api-openshift-ca" pod "c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06" (UID: "c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:24.841893 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.841883    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58181eb5-380f-49a1-bd72-b543779e9410-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58181eb5-380f-49a1-bd72-b543779e9410-maas-api-openshift-ca\") pod \"58181eb5-380f-49a1-bd72-b543779e9410\" (UID: \"58181eb5-380f-49a1-bd72-b543779e9410\") "
Oct 30 19:15:24.842022 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.841997    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4087501-1956-4f6e-8b87-9223565cce59-kube-api-access-k44c4\" (UniqueName: \"kubernetes.io/projected/a4087501-1956-4f6e-8b87-9223565cce59-kube-api-access-k44c4\") pod \"a4087501-1956-4f6e-8b87-9223565cce59\" (UID: \"a4087501-1956-4f6e-8b87-9223565cce59\") "
Oct 30 19:15:24.842308 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.842278    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58181eb5-380f-49a1-bd72-b543779e9410-maas-api-openshift-ca" pod "58181eb5-380f-49a1-bd72-b543779e9410" (UID: "58181eb5-380f-49a1-bd72-b543779e9410"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:24.842972 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.842951    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58181eb5-380f-49a1-bd72-b543779e9410-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:24.843103 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.842978    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4087501-1956-4f6e-8b87-9223565cce59-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:24.843103 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.842995    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:24.843997 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.843975    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06-maas-api-xks-ca" pod "c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06" (UID: "c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:24.844194 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.844041    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58181eb5-380f-49a1-bd72-b543779e9410-maas-api-xks-ca" pod "58181eb5-380f-49a1-bd72-b543779e9410" (UID: "58181eb5-380f-49a1-bd72-b543779e9410"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:24.844374 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.844345    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4087501-1956-4f6e-8b87-9223565cce59-maas-api-xks-ca" pod "a4087501-1956-4f6e-8b87-9223565cce59" (UID: "a4087501-1956-4f6e-8b87-9223565cce59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:24.844672 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.844653    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58181eb5-380f-49a1-bd72-b543779e9410-kube-api-access-tn757" pod "58181eb5-380f-49a1-bd72-b543779e9410" (UID: "58181eb5-380f-49a1-bd72-b543779e9410"). InnerVolumeSpecName "kube-api-access-tn757". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:24.844742 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.844680    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06-kube-api-access-k49ws" pod "c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06" (UID: "c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06"). InnerVolumeSpecName "kube-api-access-k49ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:24.844985 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.844968    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4087501-1956-4f6e-8b87-9223565cce59-kube-api-access-k44c4" pod "a4087501-1956-4f6e-8b87-9223565cce59" (UID: "a4087501-1956-4f6e-8b87-9223565cce59"). InnerVolumeSpecName "kube-api-access-k44c4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:24.944477 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.944444    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:24.944477 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.944474    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4087501-1956-4f6e-8b87-9223565cce59-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:24.944710 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.944491    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k49ws\" (UniqueName: \"kubernetes.io/projected/c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06-kube-api-access-k49ws\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:24.944710 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.944507    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58181eb5-380f-49a1-bd72-b543779e9410-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:24.944710 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.944523    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn757\" (UniqueName: \"kubernetes.io/projected/58181eb5-380f-49a1-bd72-b543779e9410-kube-api-access-tn757\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:24.944710 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:24.944536    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k44c4\" (UniqueName: \"kubernetes.io/projected/a4087501-1956-4f6e-8b87-9223565cce59-kube-api-access-k44c4\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:25.470341 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:25.470303    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6"]
Oct 30 19:15:25.535414 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:25.535384    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb"]
Oct 30 19:15:25.577136 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:25.577106    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq"
Oct 30 19:15:25.577321 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:25.577294    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq" event={"ID":"58181eb5-380f-49a1-bd72-b543779e9410","Type":"ContainerDied","Data":"bc1fc271727c5aa30555415fe9cae0ea712c60ea2b03a1bf6972ffea297f63e3"}
Oct 30 19:15:25.577384 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:25.577350    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc1fc271727c5aa30555415fe9cae0ea712c60ea2b03a1bf6972ffea297f63e3"
Oct 30 19:15:25.578994 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:25.578971    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d86012-cbvvz" event={"ID":"c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06","Type":"ContainerDied","Data":"cf7b05214be31aca7edffe26b5d30d218cec03e6edee249e80e244bc1a677fa8"}
Oct 30 19:15:25.579166 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:25.579008    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf7b05214be31aca7edffe26b5d30d218cec03e6edee249e80e244bc1a677fa8"
Oct 30 19:15:25.579166 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:25.579051    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d86012-cbvvz"
Oct 30 19:15:25.582813 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:25.582785    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ebbd68-lrwl9" event={"ID":"a4087501-1956-4f6e-8b87-9223565cce59","Type":"ContainerDied","Data":"6dc449f7c1d18cc9760065676c34c4ae1f2cfa457df4f40e78045da2754e66f6"}
Oct 30 19:15:25.582966 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:25.582823    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66ebbd68-lrwl9"
Oct 30 19:15:25.582966 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:25.582835    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dc449f7c1d18cc9760065676c34c4ae1f2cfa457df4f40e78045da2754e66f6"
Oct 30 19:15:25.840542 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:25.840432    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:26.047687 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:26.047657    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:26.235367 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:26.235335    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:26.575888 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:26.575807    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:27.012286 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:27.012250    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:27.432444 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:27.432408    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:27.810375 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:27.810292    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:28.052208 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:28.052177    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:28.252273 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:28.252238    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:28.411558 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:28.411523    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-568107-zckj8"]
Oct 30 19:15:28.457951 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:28.457916    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-568107-zckj8"]
Oct 30 19:15:28.821613 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:28.821581    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:28.836047 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:28.836018    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24e1089b-d671-45ae-8d52-607a705942cc" path="/var/lib/kubelet/pods/24e1089b-d671-45ae-8d52-607a705942cc/volumes"
Oct 30 19:15:30.470737 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:30.470678    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6" podUID="e9720622-6440-4232-8855-ae46ec5bf434" containerName="payload-processing" containerID="cri-o://a5aefc16cb3c8a3ec160196d15b320d3c3540f6ea9f9e35e6992355eb605d3ab" gracePeriod=25
Oct 30 19:15:30.535814 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:30.535747    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb" podUID="9918ebe6-aa19-488e-b259-e6201f326650" containerName="payload-pre-processing" containerID="cri-o://7ecc1ae5bfa9ec792e152a062207ab7d74b9b753618c3c9f21d203482b768de3" gracePeriod=25
Oct 30 19:15:31.503776 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:31.503746    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:31.688209 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:31.688169    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:32.520519 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.520473    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" podUID="eb82dd7d-e682-42a2-924e-a84f21172fc8" containerName="main" containerID="cri-o://bc190554447ca2a2b0675d2f7cb195b5e207fc1dc5122bd75aa59008c6ef1ab3" gracePeriod=45
Oct 30 19:15:32.553723 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.553680    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd" podUID="03ac954a-d07b-4523-8d5a-a13dffd0bd2e" containerName="main" containerID="cri-o://15805d7ed73fcef47a8b31dbe411a6bf260d0ffa4879407d4b097070f068f360" gracePeriod=45
Oct 30 19:15:32.649729 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.649701    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88fc83c8-dwnqc"]
Oct 30 19:15:32.650392 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.650373    4924 cpu_manager.go:414] "containerMap: removing container" podUID="d9aa4147-99fd-4ee4-9dd0-43ae21407002" containerName="cleanup"
Oct 30 19:15:32.650392 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.650392    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9aa4147-99fd-4ee4-9dd0-43ae21407002" containerName="cleanup"
Oct 30 19:15:32.650513 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.650410    4924 cpu_manager.go:414] "containerMap: removing container" podUID="534a63d0-c4b9-48cb-88f0-bed24a366017" containerName="cleanup"
Oct 30 19:15:32.650513 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.650416    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="534a63d0-c4b9-48cb-88f0-bed24a366017" containerName="cleanup"
Oct 30 19:15:32.650513 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.650436    4924 cpu_manager.go:414] "containerMap: removing container" podUID="142bd310-9137-4db5-a5e5-27fcf478fbb5" containerName="cleanup"
Oct 30 19:15:32.650513 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.650442    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="142bd310-9137-4db5-a5e5-27fcf478fbb5" containerName="cleanup"
Oct 30 19:15:32.650513 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.650461    4924 cpu_manager.go:414] "containerMap: removing container" podUID="e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1" containerName="cleanup"
Oct 30 19:15:32.650513 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.650468    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1" containerName="cleanup"
Oct 30 19:15:32.650513 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.650487    4924 cpu_manager.go:414] "containerMap: removing container" podUID="142bd310-9137-4db5-a5e5-27fcf478fbb5" containerName="cleanup"
Oct 30 19:15:32.650513 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.650491    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="142bd310-9137-4db5-a5e5-27fcf478fbb5" containerName="cleanup"
Oct 30 19:15:32.650513 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.650498    4924 cpu_manager.go:414] "containerMap: removing container" podUID="c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06" containerName="revoke-keys"
Oct 30 19:15:32.650513 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.650503    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06" containerName="revoke-keys"
Oct 30 19:15:32.650513 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.650515    4924 cpu_manager.go:414] "containerMap: removing container" podUID="a4087501-1956-4f6e-8b87-9223565cce59" containerName="revoke-keys"
Oct 30 19:15:32.650513 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.650519    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4087501-1956-4f6e-8b87-9223565cce59" containerName="revoke-keys"
Oct 30 19:15:32.650885 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.650527    4924 cpu_manager.go:414] "containerMap: removing container" podUID="58181eb5-380f-49a1-bd72-b543779e9410" containerName="revoke-keys"
Oct 30 19:15:32.650885 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.650532    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58181eb5-380f-49a1-bd72-b543779e9410" containerName="revoke-keys"
Oct 30 19:15:32.650885 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.650664    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="142bd310-9137-4db5-a5e5-27fcf478fbb5" containerName="cleanup"
Oct 30 19:15:32.650885 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.650674    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f86332-e4bb-4eeb-88f6-ab7c4706d6c1" containerName="cleanup"
Oct 30 19:15:32.650885 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.650686    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="142bd310-9137-4db5-a5e5-27fcf478fbb5" containerName="cleanup"
Oct 30 19:15:32.650885 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.650691    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06" containerName="revoke-keys"
Oct 30 19:15:32.650885 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.650700    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4087501-1956-4f6e-8b87-9223565cce59" containerName="revoke-keys"
Oct 30 19:15:32.650885 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.650711    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="58181eb5-380f-49a1-bd72-b543779e9410" containerName="revoke-keys"
Oct 30 19:15:32.650885 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.650723    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="142bd310-9137-4db5-a5e5-27fcf478fbb5" containerName="cleanup"
Oct 30 19:15:32.650885 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.650731    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="534a63d0-c4b9-48cb-88f0-bed24a366017" containerName="cleanup"
Oct 30 19:15:32.654262 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.654247    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fc83c8-dwnqc"
Oct 30 19:15:32.682209 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.681357    4924 generic.go:356] "Generic (PLEG): container finished" podID="03ac954a-d07b-4523-8d5a-a13dffd0bd2e" containerID="15805d7ed73fcef47a8b31dbe411a6bf260d0ffa4879407d4b097070f068f360" exitCode=0
Oct 30 19:15:32.682209 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.681601    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd" event={"ID":"03ac954a-d07b-4523-8d5a-a13dffd0bd2e","Type":"ContainerDied","Data":"15805d7ed73fcef47a8b31dbe411a6bf260d0ffa4879407d4b097070f068f360"}
Oct 30 19:15:32.683661 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.683638    4924 generic.go:356] "Generic (PLEG): container finished" podID="eb82dd7d-e682-42a2-924e-a84f21172fc8" containerID="bc190554447ca2a2b0675d2f7cb195b5e207fc1dc5122bd75aa59008c6ef1ab3" exitCode=0
Oct 30 19:15:32.683767 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.683715    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" event={"ID":"eb82dd7d-e682-42a2-924e-a84f21172fc8","Type":"ContainerDied","Data":"bc190554447ca2a2b0675d2f7cb195b5e207fc1dc5122bd75aa59008c6ef1ab3"}
Oct 30 19:15:32.705939 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.705795    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4225c\" (UniqueName: \"kubernetes.io/projected/34732c18-0104-4161-8560-5f9f32ebc273-kube-api-access-4225c\") pod \"maas-api-revoke-sub-88fc83c8-dwnqc\" (UID: \"34732c18-0104-4161-8560-5f9f32ebc273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fc83c8-dwnqc"
Oct 30 19:15:32.706108 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.705997    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34732c18-0104-4161-8560-5f9f32ebc273-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88fc83c8-dwnqc\" (UID: \"34732c18-0104-4161-8560-5f9f32ebc273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fc83c8-dwnqc"
Oct 30 19:15:32.706189 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.706131    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34732c18-0104-4161-8560-5f9f32ebc273-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88fc83c8-dwnqc\" (UID: \"34732c18-0104-4161-8560-5f9f32ebc273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fc83c8-dwnqc"
Oct 30 19:15:32.710761 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.707430    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53345e4b-s92r2"]
Oct 30 19:15:32.710761 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.708423    4924 cpu_manager.go:414] "containerMap: removing container" podUID="142bd310-9137-4db5-a5e5-27fcf478fbb5" containerName="cleanup"
Oct 30 19:15:32.710761 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.708442    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="142bd310-9137-4db5-a5e5-27fcf478fbb5" containerName="cleanup"
Oct 30 19:15:32.716700 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.716683    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88fc83c8-dwnqc"]
Oct 30 19:15:32.716857 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.716810    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53345e4b-s92r2"
Oct 30 19:15:32.749165 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.749139    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:15:32.749165 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.749163    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:15:32.752556 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.752529    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:15:32.752668 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.752649    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:15:32.778244 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.778163    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53345e4b-s92r2"]
Oct 30 19:15:32.810808 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.808210    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4225c\" (UniqueName: \"kubernetes.io/projected/34732c18-0104-4161-8560-5f9f32ebc273-kube-api-access-4225c\") pod \"maas-api-revoke-sub-88fc83c8-dwnqc\" (UID: \"34732c18-0104-4161-8560-5f9f32ebc273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fc83c8-dwnqc"
Oct 30 19:15:32.810808 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.808408    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34732c18-0104-4161-8560-5f9f32ebc273-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88fc83c8-dwnqc\" (UID: \"34732c18-0104-4161-8560-5f9f32ebc273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fc83c8-dwnqc"
Oct 30 19:15:32.810808 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.808487    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f6635c-af07-453a-85dc-4d76e40b70da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53345e4b-s92r2\" (UID: \"64f6635c-af07-453a-85dc-4d76e40b70da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53345e4b-s92r2"
Oct 30 19:15:32.810808 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.808644    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34732c18-0104-4161-8560-5f9f32ebc273-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88fc83c8-dwnqc\" (UID: \"34732c18-0104-4161-8560-5f9f32ebc273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fc83c8-dwnqc"
Oct 30 19:15:32.810808 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.808679    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f6635c-af07-453a-85dc-4d76e40b70da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53345e4b-s92r2\" (UID: \"64f6635c-af07-453a-85dc-4d76e40b70da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53345e4b-s92r2"
Oct 30 19:15:32.810808 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.808708    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6hc2\" (UniqueName: \"kubernetes.io/projected/64f6635c-af07-453a-85dc-4d76e40b70da-kube-api-access-c6hc2\") pod \"maas-api-revoke-sub-53345e4b-s92r2\" (UID: \"64f6635c-af07-453a-85dc-4d76e40b70da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53345e4b-s92r2"
Oct 30 19:15:32.810808 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.809972    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34732c18-0104-4161-8560-5f9f32ebc273-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88fc83c8-dwnqc\" (UID: \"34732c18-0104-4161-8560-5f9f32ebc273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fc83c8-dwnqc"
Oct 30 19:15:32.813778 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.813758    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34732c18-0104-4161-8560-5f9f32ebc273-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88fc83c8-dwnqc\" (UID: \"34732c18-0104-4161-8560-5f9f32ebc273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fc83c8-dwnqc"
Oct 30 19:15:32.879579 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.879552    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4225c\" (UniqueName: \"kubernetes.io/projected/34732c18-0104-4161-8560-5f9f32ebc273-kube-api-access-4225c\") pod \"maas-api-revoke-sub-88fc83c8-dwnqc\" (UID: \"34732c18-0104-4161-8560-5f9f32ebc273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fc83c8-dwnqc"
Oct 30 19:15:32.910209 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.909760    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f6635c-af07-453a-85dc-4d76e40b70da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53345e4b-s92r2\" (UID: \"64f6635c-af07-453a-85dc-4d76e40b70da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53345e4b-s92r2"
Oct 30 19:15:32.910209 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.909902    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f6635c-af07-453a-85dc-4d76e40b70da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53345e4b-s92r2\" (UID: \"64f6635c-af07-453a-85dc-4d76e40b70da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53345e4b-s92r2"
Oct 30 19:15:32.910209 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.909941    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6hc2\" (UniqueName: \"kubernetes.io/projected/64f6635c-af07-453a-85dc-4d76e40b70da-kube-api-access-c6hc2\") pod \"maas-api-revoke-sub-53345e4b-s92r2\" (UID: \"64f6635c-af07-453a-85dc-4d76e40b70da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53345e4b-s92r2"
Oct 30 19:15:32.910747 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.910572    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f6635c-af07-453a-85dc-4d76e40b70da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53345e4b-s92r2\" (UID: \"64f6635c-af07-453a-85dc-4d76e40b70da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53345e4b-s92r2"
Oct 30 19:15:32.917817 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.913311    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f6635c-af07-453a-85dc-4d76e40b70da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53345e4b-s92r2\" (UID: \"64f6635c-af07-453a-85dc-4d76e40b70da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53345e4b-s92r2"
Oct 30 19:15:32.926406 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.926377    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 19:15:32.929003 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.928960    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 19:15:32.977371 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.977317    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:32.989357 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.989324    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fc83c8-dwnqc"
Oct 30 19:15:32.992353 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:32.992315    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6hc2\" (UniqueName: \"kubernetes.io/projected/64f6635c-af07-453a-85dc-4d76e40b70da-kube-api-access-c6hc2\") pod \"maas-api-revoke-sub-53345e4b-s92r2\" (UID: \"64f6635c-af07-453a-85dc-4d76e40b70da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53345e4b-s92r2"
Oct 30 19:15:33.011025 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.010994    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-tls-certs\" (UniqueName: \"kubernetes.io/secret/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-tls-certs\") pod \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\" (UID: \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\") "
Oct 30 19:15:33.011207 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.011107    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-home\" (UniqueName: \"kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-home\") pod \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\" (UID: \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\") "
Oct 30 19:15:33.011207 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.011196    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-tmp-dir\") pod \"eb82dd7d-e682-42a2-924e-a84f21172fc8\" (UID: \"eb82dd7d-e682-42a2-924e-a84f21172fc8\") "
Oct 30 19:15:33.011330 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.011236    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-tmp-dir\") pod \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\" (UID: \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\") "
Oct 30 19:15:33.011330 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.011307    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-model-cache\") pod \"eb82dd7d-e682-42a2-924e-a84f21172fc8\" (UID: \"eb82dd7d-e682-42a2-924e-a84f21172fc8\") "
Oct 30 19:15:33.011462 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.011445    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-dshm\") pod \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\" (UID: \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\") "
Oct 30 19:15:33.011521 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.011501    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-model-cache\") pod \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\" (UID: \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\") "
Oct 30 19:15:33.011588 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.011515    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-tmp-dir" pod "eb82dd7d-e682-42a2-924e-a84f21172fc8" (UID: "eb82dd7d-e682-42a2-924e-a84f21172fc8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:33.011588 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.011534    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb82dd7d-e682-42a2-924e-a84f21172fc8-kube-api-access-wdcsm\" (UniqueName: \"kubernetes.io/projected/eb82dd7d-e682-42a2-924e-a84f21172fc8-kube-api-access-wdcsm\") pod \"eb82dd7d-e682-42a2-924e-a84f21172fc8\" (UID: \"eb82dd7d-e682-42a2-924e-a84f21172fc8\") "
Oct 30 19:15:33.011588 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.011583    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-dshm\") pod \"eb82dd7d-e682-42a2-924e-a84f21172fc8\" (UID: \"eb82dd7d-e682-42a2-924e-a84f21172fc8\") "
Oct 30 19:15:33.011753 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.011622    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-kube-api-access-pnsp8\" (UniqueName: \"kubernetes.io/projected/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-kube-api-access-pnsp8\") pod \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\" (UID: \"03ac954a-d07b-4523-8d5a-a13dffd0bd2e\") "
Oct 30 19:15:33.011753 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.011655    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-home\" (UniqueName: \"kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-home\") pod \"eb82dd7d-e682-42a2-924e-a84f21172fc8\" (UID: \"eb82dd7d-e682-42a2-924e-a84f21172fc8\") "
Oct 30 19:15:33.011753 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.011696    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb82dd7d-e682-42a2-924e-a84f21172fc8-tls-certs\" (UniqueName: \"kubernetes.io/secret/eb82dd7d-e682-42a2-924e-a84f21172fc8-tls-certs\") pod \"eb82dd7d-e682-42a2-924e-a84f21172fc8\" (UID: \"eb82dd7d-e682-42a2-924e-a84f21172fc8\") "
Oct 30 19:15:33.011753 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.011707    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-home" pod "03ac954a-d07b-4523-8d5a-a13dffd0bd2e" (UID: "03ac954a-d07b-4523-8d5a-a13dffd0bd2e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:33.012357 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.012329    4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-tmp-dir\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:33.013557 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.013330    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-model-cache" pod "eb82dd7d-e682-42a2-924e-a84f21172fc8" (UID: "eb82dd7d-e682-42a2-924e-a84f21172fc8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:33.013557 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.013340    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-model-cache" pod "03ac954a-d07b-4523-8d5a-a13dffd0bd2e" (UID: "03ac954a-d07b-4523-8d5a-a13dffd0bd2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:33.013977 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.013770    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-tmp-dir" pod "03ac954a-d07b-4523-8d5a-a13dffd0bd2e" (UID: "03ac954a-d07b-4523-8d5a-a13dffd0bd2e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:33.014821 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.014154    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-home" pod "eb82dd7d-e682-42a2-924e-a84f21172fc8" (UID: "eb82dd7d-e682-42a2-924e-a84f21172fc8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:33.018287 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.018262    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb82dd7d-e682-42a2-924e-a84f21172fc8-tls-certs" pod "eb82dd7d-e682-42a2-924e-a84f21172fc8" (UID: "eb82dd7d-e682-42a2-924e-a84f21172fc8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:33.019447 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.019249    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-dshm" pod "eb82dd7d-e682-42a2-924e-a84f21172fc8" (UID: "eb82dd7d-e682-42a2-924e-a84f21172fc8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:33.019447 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.019296    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb82dd7d-e682-42a2-924e-a84f21172fc8-kube-api-access-wdcsm" pod "eb82dd7d-e682-42a2-924e-a84f21172fc8" (UID: "eb82dd7d-e682-42a2-924e-a84f21172fc8"). InnerVolumeSpecName "kube-api-access-wdcsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:33.019447 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.019405    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-kube-api-access-pnsp8" pod "03ac954a-d07b-4523-8d5a-a13dffd0bd2e" (UID: "03ac954a-d07b-4523-8d5a-a13dffd0bd2e"). InnerVolumeSpecName "kube-api-access-pnsp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:33.020608 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.020583    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-dshm" pod "03ac954a-d07b-4523-8d5a-a13dffd0bd2e" (UID: "03ac954a-d07b-4523-8d5a-a13dffd0bd2e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:33.023417 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.023388    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-tls-certs" pod "03ac954a-d07b-4523-8d5a-a13dffd0bd2e" (UID: "03ac954a-d07b-4523-8d5a-a13dffd0bd2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:33.044315 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.044246    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53345e4b-s92r2"
Oct 30 19:15:33.094197 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.094166    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd"]
Oct 30 19:15:33.113584 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.113549    4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-tls-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:33.113584 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.113576    4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-home\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:33.113584 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.113589    4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-tmp-dir\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:33.113842 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.113597    4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-model-cache\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:33.113842 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.113605    4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-dshm\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:33.113842 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.113615    4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-model-cache\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:33.113842 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.113623    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdcsm\" (UniqueName: \"kubernetes.io/projected/eb82dd7d-e682-42a2-924e-a84f21172fc8-kube-api-access-wdcsm\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:33.113842 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.113634    4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-dshm\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:33.113842 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.113642    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnsp8\" (UniqueName: \"kubernetes.io/projected/03ac954a-d07b-4523-8d5a-a13dffd0bd2e-kube-api-access-pnsp8\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:33.113842 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.113651    4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb82dd7d-e682-42a2-924e-a84f21172fc8-home\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:33.113842 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.113658    4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb82dd7d-e682-42a2-924e-a84f21172fc8-tls-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:33.180764 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.180730    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs"]
Oct 30 19:15:33.373499 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.373468    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88fc83c8-dwnqc"]
Oct 30 19:15:33.373642 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:15:33.373473    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34732c18_0104_4161_8560_5f9f32ebc273.slice/crio-b593a57cdc07db3ed0ec7ed146e7869988af64f7f72e3e497203e5f8da4f332e WatchSource:0}: Error finding container b593a57cdc07db3ed0ec7ed146e7869988af64f7f72e3e497203e5f8da4f332e: Status 404 returned error can't find the container with id b593a57cdc07db3ed0ec7ed146e7869988af64f7f72e3e497203e5f8da4f332e
Oct 30 19:15:33.466473 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.466293    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53345e4b-s92r2"]
Oct 30 19:15:33.466898 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:15:33.466865    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f6635c_af07_453a_85dc_4d76e40b70da.slice/crio-8b39d407449618bc3e61be9ee5022a74b37cfe3daee238bca8267bb194176bdb WatchSource:0}: Error finding container 8b39d407449618bc3e61be9ee5022a74b37cfe3daee238bca8267bb194176bdb: Status 404 returned error can't find the container with id 8b39d407449618bc3e61be9ee5022a74b37cfe3daee238bca8267bb194176bdb
Oct 30 19:15:33.500275 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.498355    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:33.680386 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.680295    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:33.692830 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.692804    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"
Oct 30 19:15:33.693008 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.692842    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" event={"ID":"eb82dd7d-e682-42a2-924e-a84f21172fc8","Type":"ContainerDied","Data":"e1ff55590e369e85ea46103efb907a1c23e8b782fff4b8b01aa32ecccbb12466"}
Oct 30 19:15:33.693008 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.692895    4924 scope.go:122] "RemoveContainer" containerID="bc190554447ca2a2b0675d2f7cb195b5e207fc1dc5122bd75aa59008c6ef1ab3"
Oct 30 19:15:33.694516 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.694500    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"
Oct 30 19:15:33.694648 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.694544    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd" event={"ID":"03ac954a-d07b-4523-8d5a-a13dffd0bd2e","Type":"ContainerDied","Data":"d0e297abd8251ffd89ba690a4071f7b0a3d36e2b89b94360324e864b6483f820"}
Oct 30 19:15:33.696003 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.695980    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53345e4b-s92r2" event={"ID":"64f6635c-af07-453a-85dc-4d76e40b70da","Type":"ContainerStarted","Data":"8b39d407449618bc3e61be9ee5022a74b37cfe3daee238bca8267bb194176bdb"}
Oct 30 19:15:33.697579 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.697552    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fc83c8-dwnqc" event={"ID":"34732c18-0104-4161-8560-5f9f32ebc273","Type":"ContainerStarted","Data":"b593a57cdc07db3ed0ec7ed146e7869988af64f7f72e3e497203e5f8da4f332e"}
Oct 30 19:15:33.704781 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.704764    4924 scope.go:122] "RemoveContainer" containerID="15805d7ed73fcef47a8b31dbe411a6bf260d0ffa4879407d4b097070f068f360"
Oct 30 19:15:33.849551 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.849523    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"]
Oct 30 19:15:33.856382 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.856346    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5f80ac/simulator-w1-5f80ac-kserve-69cdd574df-2vfzd"]
Oct 30 19:15:33.924940 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.924909    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6" podUID="eb82dd7d-e682-42a2-924e-a84f21172fc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded"
Oct 30 19:15:33.960546 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.960463    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"]
Oct 30 19:15:33.966948 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:33.966924    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5f80ac/distinct-2-w1-5f80ac-kserve-79c884d644-dn4q6"]
Oct 30 19:15:34.725350 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:34.725311    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fc83c8-dwnqc" event={"ID":"34732c18-0104-4161-8560-5f9f32ebc273","Type":"ContainerStarted","Data":"3d3ed449af66eecea3164a9e6c2afa82b207f1ca5b430207360288935c5744ca"}
Oct 30 19:15:34.729436 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:34.729409    4924 generic.go:356] "Generic (PLEG): container finished" podID="64f6635c-af07-453a-85dc-4d76e40b70da" containerID="0b18634c91f2b5568c37c7f37fc68e58f4b0b2cc4e49be27435a37ca33434e14" exitCode=0
Oct 30 19:15:34.729562 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:34.729485    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53345e4b-s92r2" event={"ID":"64f6635c-af07-453a-85dc-4d76e40b70da","Type":"ContainerDied","Data":"0b18634c91f2b5568c37c7f37fc68e58f4b0b2cc4e49be27435a37ca33434e14"}
Oct 30 19:15:34.863175 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:34.860691    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03ac954a-d07b-4523-8d5a-a13dffd0bd2e" path="/var/lib/kubelet/pods/03ac954a-d07b-4523-8d5a-a13dffd0bd2e/volumes"
Oct 30 19:15:34.863175 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:34.861441    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb82dd7d-e682-42a2-924e-a84f21172fc8" path="/var/lib/kubelet/pods/eb82dd7d-e682-42a2-924e-a84f21172fc8/volumes"
Oct 30 19:15:35.241695 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:35.241605    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:35.428791 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:35.428755    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:35.628709 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:35.628671    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:35.739495 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:35.739450    4924 generic.go:356] "Generic (PLEG): container finished" podID="34732c18-0104-4161-8560-5f9f32ebc273" containerID="3d3ed449af66eecea3164a9e6c2afa82b207f1ca5b430207360288935c5744ca" exitCode=0
Oct 30 19:15:35.739918 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:35.739491    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fc83c8-dwnqc" event={"ID":"34732c18-0104-4161-8560-5f9f32ebc273","Type":"ContainerDied","Data":"3d3ed449af66eecea3164a9e6c2afa82b207f1ca5b430207360288935c5744ca"}
Oct 30 19:15:35.790394 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:35.790354    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9"]
Oct 30 19:15:35.838117 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:35.838088    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2ef49a-ghnf9"]
Oct 30 19:15:35.962608 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:35.962583    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53345e4b-s92r2"
Oct 30 19:15:35.964714 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:35.964696    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fc83c8-dwnqc"
Oct 30 19:15:36.133992 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.133953    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64f6635c-af07-453a-85dc-4d76e40b70da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f6635c-af07-453a-85dc-4d76e40b70da-maas-api-openshift-ca\") pod \"64f6635c-af07-453a-85dc-4d76e40b70da\" (UID: \"64f6635c-af07-453a-85dc-4d76e40b70da\") "
Oct 30 19:15:36.134216 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.134090    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34732c18-0104-4161-8560-5f9f32ebc273-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34732c18-0104-4161-8560-5f9f32ebc273-maas-api-openshift-ca\") pod \"34732c18-0104-4161-8560-5f9f32ebc273\" (UID: \"34732c18-0104-4161-8560-5f9f32ebc273\") "
Oct 30 19:15:36.134216 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.134131    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64f6635c-af07-453a-85dc-4d76e40b70da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f6635c-af07-453a-85dc-4d76e40b70da-maas-api-xks-ca\") pod \"64f6635c-af07-453a-85dc-4d76e40b70da\" (UID: \"64f6635c-af07-453a-85dc-4d76e40b70da\") "
Oct 30 19:15:36.134329 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.134294    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64f6635c-af07-453a-85dc-4d76e40b70da-kube-api-access-c6hc2\" (UniqueName: \"kubernetes.io/projected/64f6635c-af07-453a-85dc-4d76e40b70da-kube-api-access-c6hc2\") pod \"64f6635c-af07-453a-85dc-4d76e40b70da\" (UID: \"64f6635c-af07-453a-85dc-4d76e40b70da\") "
Oct 30 19:15:36.134389 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.134359    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34732c18-0104-4161-8560-5f9f32ebc273-kube-api-access-4225c\" (UniqueName: \"kubernetes.io/projected/34732c18-0104-4161-8560-5f9f32ebc273-kube-api-access-4225c\") pod \"34732c18-0104-4161-8560-5f9f32ebc273\" (UID: \"34732c18-0104-4161-8560-5f9f32ebc273\") "
Oct 30 19:15:36.134447 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.134405    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f6635c-af07-453a-85dc-4d76e40b70da-maas-api-openshift-ca" pod "64f6635c-af07-453a-85dc-4d76e40b70da" (UID: "64f6635c-af07-453a-85dc-4d76e40b70da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:36.134519 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.134447    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34732c18-0104-4161-8560-5f9f32ebc273-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34732c18-0104-4161-8560-5f9f32ebc273-maas-api-xks-ca\") pod \"34732c18-0104-4161-8560-5f9f32ebc273\" (UID: \"34732c18-0104-4161-8560-5f9f32ebc273\") "
Oct 30 19:15:36.134604 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.134585    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34732c18-0104-4161-8560-5f9f32ebc273-maas-api-openshift-ca" pod "34732c18-0104-4161-8560-5f9f32ebc273" (UID: "34732c18-0104-4161-8560-5f9f32ebc273"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:36.135211 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.135185    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34732c18-0104-4161-8560-5f9f32ebc273-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:36.135354 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.135212    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f6635c-af07-453a-85dc-4d76e40b70da-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:36.136608 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.136581    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34732c18-0104-4161-8560-5f9f32ebc273-maas-api-xks-ca" pod "34732c18-0104-4161-8560-5f9f32ebc273" (UID: "34732c18-0104-4161-8560-5f9f32ebc273"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:36.136739 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.136721    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64f6635c-af07-453a-85dc-4d76e40b70da-maas-api-xks-ca" pod "64f6635c-af07-453a-85dc-4d76e40b70da" (UID: "64f6635c-af07-453a-85dc-4d76e40b70da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:36.136914 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.136882    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f6635c-af07-453a-85dc-4d76e40b70da-kube-api-access-c6hc2" pod "64f6635c-af07-453a-85dc-4d76e40b70da" (UID: "64f6635c-af07-453a-85dc-4d76e40b70da"). InnerVolumeSpecName "kube-api-access-c6hc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:36.136914 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.136902    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34732c18-0104-4161-8560-5f9f32ebc273-kube-api-access-4225c" pod "34732c18-0104-4161-8560-5f9f32ebc273" (UID: "34732c18-0104-4161-8560-5f9f32ebc273"). InnerVolumeSpecName "kube-api-access-4225c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:36.236782 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.236705    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34732c18-0104-4161-8560-5f9f32ebc273-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:36.236782 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.236734    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f6635c-af07-453a-85dc-4d76e40b70da-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:36.236782 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.236743    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6hc2\" (UniqueName: \"kubernetes.io/projected/64f6635c-af07-453a-85dc-4d76e40b70da-kube-api-access-c6hc2\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:36.236782 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.236754    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4225c\" (UniqueName: \"kubernetes.io/projected/34732c18-0104-4161-8560-5f9f32ebc273-kube-api-access-4225c\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:36.282515 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.282483    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:36.468050 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.468022    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:36.639799 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.639761    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"]
Oct 30 19:15:36.640050 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.640021    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg" podUID="c01a04de-05cf-4bcc-83d3-a7d51ffc9977" containerName="istio-proxy" containerID="cri-o://7a0eae0e04271c49494f654b88f1a5377ef0e49822daeafc2ca00a391bde6510" gracePeriod=30
Oct 30 19:15:36.728277 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.728247    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:36.826700 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.826666    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fc83c8-dwnqc"
Oct 30 19:15:36.827321 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.827235    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fc83c8-dwnqc" event={"ID":"34732c18-0104-4161-8560-5f9f32ebc273","Type":"ContainerDied","Data":"b593a57cdc07db3ed0ec7ed146e7869988af64f7f72e3e497203e5f8da4f332e"}
Oct 30 19:15:36.827321 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.827285    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b593a57cdc07db3ed0ec7ed146e7869988af64f7f72e3e497203e5f8da4f332e"
Oct 30 19:15:36.846012 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.845985    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53345e4b-s92r2"
Oct 30 19:15:36.850090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.848132    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36ced340-da28-45e1-867f-de3d1ff4017a" path="/var/lib/kubelet/pods/36ced340-da28-45e1-867f-de3d1ff4017a/volumes"
Oct 30 19:15:36.850090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.848956    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53345e4b-s92r2" event={"ID":"64f6635c-af07-453a-85dc-4d76e40b70da","Type":"ContainerDied","Data":"8b39d407449618bc3e61be9ee5022a74b37cfe3daee238bca8267bb194176bdb"}
Oct 30 19:15:36.850090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:36.849004    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b39d407449618bc3e61be9ee5022a74b37cfe3daee238bca8267bb194176bdb"
Oct 30 19:15:38.095368 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.095326    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd" podUID="124af7f9-61f7-47d4-b932-796d501d85cb" containerName="payload-processing" containerID="cri-o://92e8c8681abf62030001bce7dec7393e2fa099af0eb1672d7173bdf58aa6daba" gracePeriod=25
Oct 30 19:15:38.181397 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.181358    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs" podUID="fac381e9-2afe-4ea7-9b09-b35953682bec" containerName="payload-pre-processing" containerID="cri-o://0371d94615143ba5fc087a3ec18b443e29d34a4566c69e3a180516e8fdc14e5b" gracePeriod=25
Oct 30 19:15:38.405213 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.405176    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6" podUID="e9720622-6440-4232-8855-ae46ec5bf434" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:15:38.405213 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.405204    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb" podUID="9918ebe6-aa19-488e-b259-e6201f326650" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:15:38.452682 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.452654    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd"
Oct 30 19:15:38.487949 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.487926    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs"
Oct 30 19:15:38.569208 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.569179    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/124af7f9-61f7-47d4-b932-796d501d85cb-kube-api-access-5r6zt\" (UniqueName: \"kubernetes.io/projected/124af7f9-61f7-47d4-b932-796d501d85cb-kube-api-access-5r6zt\") pod \"124af7f9-61f7-47d4-b932-796d501d85cb\" (UID: \"124af7f9-61f7-47d4-b932-796d501d85cb\") "
Oct 30 19:15:38.569472 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.569217    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fac381e9-2afe-4ea7-9b09-b35953682bec-kube-api-access-6kjtd\" (UniqueName: \"kubernetes.io/projected/fac381e9-2afe-4ea7-9b09-b35953682bec-kube-api-access-6kjtd\") pod \"fac381e9-2afe-4ea7-9b09-b35953682bec\" (UID: \"fac381e9-2afe-4ea7-9b09-b35953682bec\") "
Oct 30 19:15:38.569472 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.569327    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/124af7f9-61f7-47d4-b932-796d501d85cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/124af7f9-61f7-47d4-b932-796d501d85cb-plugins-config-volume\") pod \"124af7f9-61f7-47d4-b932-796d501d85cb\" (UID: \"124af7f9-61f7-47d4-b932-796d501d85cb\") "
Oct 30 19:15:38.569472 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.569431    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fac381e9-2afe-4ea7-9b09-b35953682bec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac381e9-2afe-4ea7-9b09-b35953682bec-plugins-config-volume\") pod \"fac381e9-2afe-4ea7-9b09-b35953682bec\" (UID: \"fac381e9-2afe-4ea7-9b09-b35953682bec\") "
Oct 30 19:15:38.570278 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.570250    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fac381e9-2afe-4ea7-9b09-b35953682bec-plugins-config-volume" pod "fac381e9-2afe-4ea7-9b09-b35953682bec" (UID: "fac381e9-2afe-4ea7-9b09-b35953682bec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:38.570600 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.570581    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/124af7f9-61f7-47d4-b932-796d501d85cb-plugins-config-volume" pod "124af7f9-61f7-47d4-b932-796d501d85cb" (UID: "124af7f9-61f7-47d4-b932-796d501d85cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:38.572222 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.572193    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac381e9-2afe-4ea7-9b09-b35953682bec-kube-api-access-6kjtd" pod "fac381e9-2afe-4ea7-9b09-b35953682bec" (UID: "fac381e9-2afe-4ea7-9b09-b35953682bec"). InnerVolumeSpecName "kube-api-access-6kjtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:38.572537 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.572519    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124af7f9-61f7-47d4-b932-796d501d85cb-kube-api-access-5r6zt" pod "124af7f9-61f7-47d4-b932-796d501d85cb" (UID: "124af7f9-61f7-47d4-b932-796d501d85cb"). InnerVolumeSpecName "kube-api-access-5r6zt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:38.654204 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.654126    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:38.672077 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.672030    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r6zt\" (UniqueName: \"kubernetes.io/projected/124af7f9-61f7-47d4-b932-796d501d85cb-kube-api-access-5r6zt\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:38.672230 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.672086    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kjtd\" (UniqueName: \"kubernetes.io/projected/fac381e9-2afe-4ea7-9b09-b35953682bec-kube-api-access-6kjtd\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:38.672230 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.672103    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/124af7f9-61f7-47d4-b932-796d501d85cb-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:38.672230 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.672118    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac381e9-2afe-4ea7-9b09-b35953682bec-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:38.862199 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.862173    4924 generic.go:356] "Generic (PLEG): container finished" podID="124af7f9-61f7-47d4-b932-796d501d85cb" containerID="92e8c8681abf62030001bce7dec7393e2fa099af0eb1672d7173bdf58aa6daba" exitCode=0
Oct 30 19:15:38.862334 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.862216    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd"
Oct 30 19:15:38.862334 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.862255    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd" event={"ID":"124af7f9-61f7-47d4-b932-796d501d85cb","Type":"ContainerDied","Data":"92e8c8681abf62030001bce7dec7393e2fa099af0eb1672d7173bdf58aa6daba"}
Oct 30 19:15:38.862334 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.862308    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd" event={"ID":"124af7f9-61f7-47d4-b932-796d501d85cb","Type":"ContainerDied","Data":"10dd8870a6a9a3e1ba3e4d8be72da166b0cd25e49264b4ad81d7757f78f9af44"}
Oct 30 19:15:38.862565 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.862340    4924 scope.go:122] "RemoveContainer" containerID="92e8c8681abf62030001bce7dec7393e2fa099af0eb1672d7173bdf58aa6daba"
Oct 30 19:15:38.863863 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.863831    4924 generic.go:356] "Generic (PLEG): container finished" podID="fac381e9-2afe-4ea7-9b09-b35953682bec" containerID="0371d94615143ba5fc087a3ec18b443e29d34a4566c69e3a180516e8fdc14e5b" exitCode=0
Oct 30 19:15:38.863966 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.863924    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs" event={"ID":"fac381e9-2afe-4ea7-9b09-b35953682bec","Type":"ContainerDied","Data":"0371d94615143ba5fc087a3ec18b443e29d34a4566c69e3a180516e8fdc14e5b"}
Oct 30 19:15:38.864029 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.863966    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs" event={"ID":"fac381e9-2afe-4ea7-9b09-b35953682bec","Type":"ContainerDied","Data":"0348c785ea40ca5c9e9a80f8a3daddfcc1b72112092b850117c36ccd9ab6a157"}
Oct 30 19:15:38.864140 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.864030    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs"
Oct 30 19:15:38.875815 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.875798    4924 scope.go:122] "RemoveContainer" containerID="92e8c8681abf62030001bce7dec7393e2fa099af0eb1672d7173bdf58aa6daba"
Oct 30 19:15:38.876107 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:38.876085    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92e8c8681abf62030001bce7dec7393e2fa099af0eb1672d7173bdf58aa6daba\": container with ID starting with 92e8c8681abf62030001bce7dec7393e2fa099af0eb1672d7173bdf58aa6daba not found: ID does not exist" containerID="92e8c8681abf62030001bce7dec7393e2fa099af0eb1672d7173bdf58aa6daba"
Oct 30 19:15:38.876170 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.876113    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92e8c8681abf62030001bce7dec7393e2fa099af0eb1672d7173bdf58aa6daba"} err="failed to get container status \"92e8c8681abf62030001bce7dec7393e2fa099af0eb1672d7173bdf58aa6daba\": rpc error: code = NotFound desc = could not find container \"92e8c8681abf62030001bce7dec7393e2fa099af0eb1672d7173bdf58aa6daba\": container with ID starting with 92e8c8681abf62030001bce7dec7393e2fa099af0eb1672d7173bdf58aa6daba not found: ID does not exist"
Oct 30 19:15:38.876170 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.876129    4924 scope.go:122] "RemoveContainer" containerID="0371d94615143ba5fc087a3ec18b443e29d34a4566c69e3a180516e8fdc14e5b"
Oct 30 19:15:38.892482 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.892101    4924 scope.go:122] "RemoveContainer" containerID="0371d94615143ba5fc087a3ec18b443e29d34a4566c69e3a180516e8fdc14e5b"
Oct 30 19:15:38.892482 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:38.892376    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0371d94615143ba5fc087a3ec18b443e29d34a4566c69e3a180516e8fdc14e5b\": container with ID starting with 0371d94615143ba5fc087a3ec18b443e29d34a4566c69e3a180516e8fdc14e5b not found: ID does not exist" containerID="0371d94615143ba5fc087a3ec18b443e29d34a4566c69e3a180516e8fdc14e5b"
Oct 30 19:15:38.892482 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:38.892406    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0371d94615143ba5fc087a3ec18b443e29d34a4566c69e3a180516e8fdc14e5b"} err="failed to get container status \"0371d94615143ba5fc087a3ec18b443e29d34a4566c69e3a180516e8fdc14e5b\": rpc error: code = NotFound desc = could not find container \"0371d94615143ba5fc087a3ec18b443e29d34a4566c69e3a180516e8fdc14e5b\": container with ID starting with 0371d94615143ba5fc087a3ec18b443e29d34a4566c69e3a180516e8fdc14e5b not found: ID does not exist"
Oct 30 19:15:39.001473 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.001371    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:39.024274 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.024241    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs"]
Oct 30 19:15:39.041126 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.040977    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8e31a7-57bccf59ff-gzwcs"]
Oct 30 19:15:39.137286 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.137257    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd"]
Oct 30 19:15:39.144158 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.144128    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8e31a7-57dd67f87d-s2gnd"]
Oct 30 19:15:39.186502 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.186472    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:39.309342 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.309264    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n"]
Oct 30 19:15:39.309866 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.309852    4924 cpu_manager.go:414] "containerMap: removing container" podUID="64f6635c-af07-453a-85dc-4d76e40b70da" containerName="revoke-keys"
Oct 30 19:15:39.309938 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.309867    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64f6635c-af07-453a-85dc-4d76e40b70da" containerName="revoke-keys"
Oct 30 19:15:39.309938 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.309880    4924 cpu_manager.go:414] "containerMap: removing container" podUID="34732c18-0104-4161-8560-5f9f32ebc273" containerName="revoke-keys"
Oct 30 19:15:39.309938 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.309887    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34732c18-0104-4161-8560-5f9f32ebc273" containerName="revoke-keys"
Oct 30 19:15:39.309938 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.309910    4924 cpu_manager.go:414] "containerMap: removing container" podUID="fac381e9-2afe-4ea7-9b09-b35953682bec" containerName="payload-pre-processing"
Oct 30 19:15:39.309938 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.309915    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fac381e9-2afe-4ea7-9b09-b35953682bec" containerName="payload-pre-processing"
Oct 30 19:15:39.309938 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.309925    4924 cpu_manager.go:414] "containerMap: removing container" podUID="eb82dd7d-e682-42a2-924e-a84f21172fc8" containerName="main"
Oct 30 19:15:39.309938 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.309929    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb82dd7d-e682-42a2-924e-a84f21172fc8" containerName="main"
Oct 30 19:15:39.309938 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.309937    4924 cpu_manager.go:414] "containerMap: removing container" podUID="124af7f9-61f7-47d4-b932-796d501d85cb" containerName="payload-processing"
Oct 30 19:15:39.309938 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.309941    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="124af7f9-61f7-47d4-b932-796d501d85cb" containerName="payload-processing"
Oct 30 19:15:39.310267 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.309966    4924 cpu_manager.go:414] "containerMap: removing container" podUID="03ac954a-d07b-4523-8d5a-a13dffd0bd2e" containerName="main"
Oct 30 19:15:39.310267 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.309971    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ac954a-d07b-4523-8d5a-a13dffd0bd2e" containerName="main"
Oct 30 19:15:39.310267 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.310054    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="34732c18-0104-4161-8560-5f9f32ebc273" containerName="revoke-keys"
Oct 30 19:15:39.310267 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.310082    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="fac381e9-2afe-4ea7-9b09-b35953682bec" containerName="payload-pre-processing"
Oct 30 19:15:39.310267 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.310095    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb82dd7d-e682-42a2-924e-a84f21172fc8" containerName="main"
Oct 30 19:15:39.310267 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.310103    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="124af7f9-61f7-47d4-b932-796d501d85cb" containerName="payload-processing"
Oct 30 19:15:39.310267 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.310117    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ac954a-d07b-4523-8d5a-a13dffd0bd2e" containerName="main"
Oct 30 19:15:39.310267 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.310125    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="64f6635c-af07-453a-85dc-4d76e40b70da" containerName="revoke-keys"
Oct 30 19:15:39.315254 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.315236    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n"
Oct 30 19:15:39.369857 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.369733    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:15:39.370016 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.369892    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:15:39.370425 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.370391    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:15:39.370534 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.370496    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:15:39.374998 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.374973    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n"]
Oct 30 19:15:39.384982 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.384950    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cab88e32-0265-482d-b3f1-a2e33f1dd3ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n\" (UID: \"cab88e32-0265-482d-b3f1-a2e33f1dd3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n"
Oct 30 19:15:39.385127 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.384995    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj7zf\" (UniqueName: \"kubernetes.io/projected/cab88e32-0265-482d-b3f1-a2e33f1dd3ac-kube-api-access-dj7zf\") pod \"maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n\" (UID: \"cab88e32-0265-482d-b3f1-a2e33f1dd3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n"
Oct 30 19:15:39.385356 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.385333    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cab88e32-0265-482d-b3f1-a2e33f1dd3ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n\" (UID: \"cab88e32-0265-482d-b3f1-a2e33f1dd3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n"
Oct 30 19:15:39.500109 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.487714    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cab88e32-0265-482d-b3f1-a2e33f1dd3ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n\" (UID: \"cab88e32-0265-482d-b3f1-a2e33f1dd3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n"
Oct 30 19:15:39.500109 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.487815    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cab88e32-0265-482d-b3f1-a2e33f1dd3ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n\" (UID: \"cab88e32-0265-482d-b3f1-a2e33f1dd3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n"
Oct 30 19:15:39.500109 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.487847    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj7zf\" (UniqueName: \"kubernetes.io/projected/cab88e32-0265-482d-b3f1-a2e33f1dd3ac-kube-api-access-dj7zf\") pod \"maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n\" (UID: \"cab88e32-0265-482d-b3f1-a2e33f1dd3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n"
Oct 30 19:15:39.500109 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.489522    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cab88e32-0265-482d-b3f1-a2e33f1dd3ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n\" (UID: \"cab88e32-0265-482d-b3f1-a2e33f1dd3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n"
Oct 30 19:15:39.513572 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.513537    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cab88e32-0265-482d-b3f1-a2e33f1dd3ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n\" (UID: \"cab88e32-0265-482d-b3f1-a2e33f1dd3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n"
Oct 30 19:15:39.541948 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.541919    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj7zf\" (UniqueName: \"kubernetes.io/projected/cab88e32-0265-482d-b3f1-a2e33f1dd3ac-kube-api-access-dj7zf\") pod \"maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n\" (UID: \"cab88e32-0265-482d-b3f1-a2e33f1dd3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n"
Oct 30 19:15:39.629746 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.629714    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n"
Oct 30 19:15:39.910023 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:39.909941    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:40.020086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:40.018765    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n"]
Oct 30 19:15:40.027813 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:15:40.027781    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab88e32_0265_482d_b3f1_a2e33f1dd3ac.slice/crio-86e1808c11caf78241caec246396974764f2634abbbe7f9a10d60fda2680b9e3 WatchSource:0}: Error finding container 86e1808c11caf78241caec246396974764f2634abbbe7f9a10d60fda2680b9e3: Status 404 returned error can't find the container with id 86e1808c11caf78241caec246396974764f2634abbbe7f9a10d60fda2680b9e3
Oct 30 19:15:40.739407 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:40.739378    4924 scope.go:122] "RemoveContainer" containerID="83ce4836ff6d9b028d1de692a5dd50735756e3d75794d6aa1140a41dc52c9ffb"
Oct 30 19:15:40.910009 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:40.909974    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="124af7f9-61f7-47d4-b932-796d501d85cb" path="/var/lib/kubelet/pods/124af7f9-61f7-47d4-b932-796d501d85cb/volumes"
Oct 30 19:15:40.910808 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:40.910784    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fac381e9-2afe-4ea7-9b09-b35953682bec" path="/var/lib/kubelet/pods/fac381e9-2afe-4ea7-9b09-b35953682bec/volumes"
Oct 30 19:15:40.965528 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:40.965494    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n" event={"ID":"cab88e32-0265-482d-b3f1-a2e33f1dd3ac","Type":"ContainerStarted","Data":"9decf978143b0f918adb02392bedf16ab768273b91f846999894c496760ddf60"}
Oct 30 19:15:40.965632 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:40.965556    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n" event={"ID":"cab88e32-0265-482d-b3f1-a2e33f1dd3ac","Type":"ContainerStarted","Data":"86e1808c11caf78241caec246396974764f2634abbbe7f9a10d60fda2680b9e3"}
Oct 30 19:15:41.082227 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:41.082176    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n" podStartSLOduration=2.082160953 podStartE2EDuration="2.082160953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:41.080043217 +0000 UTC m=+2342.357093659" watchObservedRunningTime="2026-10-30 19:15:41.082160953 +0000 UTC m=+2342.359211398"
Oct 30 19:15:41.381684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:41.381653    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:41.596120 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:41.596088    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:41.930007 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:41.929980    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:41.936764 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:41.936743    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:15:42.001216 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:41.997385    4924 generic.go:356] "Generic (PLEG): container finished" podID="cab88e32-0265-482d-b3f1-a2e33f1dd3ac" containerID="9decf978143b0f918adb02392bedf16ab768273b91f846999894c496760ddf60" exitCode=0
Oct 30 19:15:42.001216 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:41.997611    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n" event={"ID":"cab88e32-0265-482d-b3f1-a2e33f1dd3ac","Type":"ContainerDied","Data":"9decf978143b0f918adb02392bedf16ab768273b91f846999894c496760ddf60"}
Oct 30 19:15:42.001216 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:41.999796    4924 generic.go:356] "Generic (PLEG): container finished" podID="c01a04de-05cf-4bcc-83d3-a7d51ffc9977" containerID="7a0eae0e04271c49494f654b88f1a5377ef0e49822daeafc2ca00a391bde6510" exitCode=0
Oct 30 19:15:42.001216 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:41.999848    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg" event={"ID":"c01a04de-05cf-4bcc-83d3-a7d51ffc9977","Type":"ContainerDied","Data":"7a0eae0e04271c49494f654b88f1a5377ef0e49822daeafc2ca00a391bde6510"}
Oct 30 19:15:42.001216 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:41.999881    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg" event={"ID":"c01a04de-05cf-4bcc-83d3-a7d51ffc9977","Type":"ContainerDied","Data":"0cd9d1a914a595113aad2dd490b99af37ebdb0981f08412757eb9280af0a0ca7"}
Oct 30 19:15:42.001216 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:41.999905    4924 scope.go:122] "RemoveContainer" containerID="7a0eae0e04271c49494f654b88f1a5377ef0e49822daeafc2ca00a391bde6510"
Oct 30 19:15:42.001216 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.000042    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"
Oct 30 19:15:42.014404 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.014380    4924 scope.go:122] "RemoveContainer" containerID="53aafde06247443f6080f5aa3cbe2add276430dd401b6d7a77c7d23da472b5e0"
Oct 30 19:15:42.024733 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.024709    4924 scope.go:122] "RemoveContainer" containerID="7a0eae0e04271c49494f654b88f1a5377ef0e49822daeafc2ca00a391bde6510"
Oct 30 19:15:42.025042 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:42.025012    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a0eae0e04271c49494f654b88f1a5377ef0e49822daeafc2ca00a391bde6510\": container with ID starting with 7a0eae0e04271c49494f654b88f1a5377ef0e49822daeafc2ca00a391bde6510 not found: ID does not exist" containerID="7a0eae0e04271c49494f654b88f1a5377ef0e49822daeafc2ca00a391bde6510"
Oct 30 19:15:42.025170 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.025043    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a0eae0e04271c49494f654b88f1a5377ef0e49822daeafc2ca00a391bde6510"} err="failed to get container status \"7a0eae0e04271c49494f654b88f1a5377ef0e49822daeafc2ca00a391bde6510\": rpc error: code = NotFound desc = could not find container \"7a0eae0e04271c49494f654b88f1a5377ef0e49822daeafc2ca00a391bde6510\": container with ID starting with 7a0eae0e04271c49494f654b88f1a5377ef0e49822daeafc2ca00a391bde6510 not found: ID does not exist"
Oct 30 19:15:42.025170 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.025080    4924 scope.go:122] "RemoveContainer" containerID="53aafde06247443f6080f5aa3cbe2add276430dd401b6d7a77c7d23da472b5e0"
Oct 30 19:15:42.025357 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:42.025340    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53aafde06247443f6080f5aa3cbe2add276430dd401b6d7a77c7d23da472b5e0\": container with ID starting with 53aafde06247443f6080f5aa3cbe2add276430dd401b6d7a77c7d23da472b5e0 not found: ID does not exist" containerID="53aafde06247443f6080f5aa3cbe2add276430dd401b6d7a77c7d23da472b5e0"
Oct 30 19:15:42.025411 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.025364    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53aafde06247443f6080f5aa3cbe2add276430dd401b6d7a77c7d23da472b5e0"} err="failed to get container status \"53aafde06247443f6080f5aa3cbe2add276430dd401b6d7a77c7d23da472b5e0\": rpc error: code = NotFound desc = could not find container \"53aafde06247443f6080f5aa3cbe2add276430dd401b6d7a77c7d23da472b5e0\": container with ID starting with 53aafde06247443f6080f5aa3cbe2add276430dd401b6d7a77c7d23da472b5e0 not found: ID does not exist"
Oct 30 19:15:42.026730 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.026711    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-token\" (UniqueName: \"kubernetes.io/projected/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-token\") pod \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") "
Oct 30 19:15:42.026823 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.026810    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istiod-ca-cert\") pod \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") "
Oct 30 19:15:42.026862 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.026846    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-workload-certs\") pod \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") "
Oct 30 19:15:42.026913 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.026886    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-podinfo\") pod \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") "
Oct 30 19:15:42.026979 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.026954    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-credential-socket\") pod \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") "
Oct 30 19:15:42.027048 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.027010    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-workload-socket\") pod \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") "
Oct 30 19:15:42.027146 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.027099    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-data\") pod \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") "
Oct 30 19:15:42.027207 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.027148    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-workload-certs" pod "c01a04de-05cf-4bcc-83d3-a7d51ffc9977" (UID: "c01a04de-05cf-4bcc-83d3-a7d51ffc9977"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:42.027207 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.027185    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-kube-api-access-88w66\" (UniqueName: \"kubernetes.io/projected/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-kube-api-access-88w66\") pod \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") "
Oct 30 19:15:42.027315 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.027243    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-credential-socket" pod "c01a04de-05cf-4bcc-83d3-a7d51ffc9977" (UID: "c01a04de-05cf-4bcc-83d3-a7d51ffc9977"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:42.027315 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.027273    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-workload-socket" pod "c01a04de-05cf-4bcc-83d3-a7d51ffc9977" (UID: "c01a04de-05cf-4bcc-83d3-a7d51ffc9977"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:42.027427 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.027313    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-envoy\") pod \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\" (UID: \"c01a04de-05cf-4bcc-83d3-a7d51ffc9977\") "
Oct 30 19:15:42.027427 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.027314    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istiod-ca-cert" pod "c01a04de-05cf-4bcc-83d3-a7d51ffc9977" (UID: "c01a04de-05cf-4bcc-83d3-a7d51ffc9977"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:42.027523 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.027363    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-data" pod "c01a04de-05cf-4bcc-83d3-a7d51ffc9977" (UID: "c01a04de-05cf-4bcc-83d3-a7d51ffc9977"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:42.027878 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.027861    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:42.027973 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.027894    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:42.027973 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.027908    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:42.027973 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.027928    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:42.027973 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.027942    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:42.029307 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.029282    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-token" pod "c01a04de-05cf-4bcc-83d3-a7d51ffc9977" (UID: "c01a04de-05cf-4bcc-83d3-a7d51ffc9977"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:42.029388 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.029342    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-envoy" pod "c01a04de-05cf-4bcc-83d3-a7d51ffc9977" (UID: "c01a04de-05cf-4bcc-83d3-a7d51ffc9977"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:42.029600 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.029583    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-kube-api-access-88w66" pod "c01a04de-05cf-4bcc-83d3-a7d51ffc9977" (UID: "c01a04de-05cf-4bcc-83d3-a7d51ffc9977"). InnerVolumeSpecName "kube-api-access-88w66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:42.029657 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.029630    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-podinfo" pod "c01a04de-05cf-4bcc-83d3-a7d51ffc9977" (UID: "c01a04de-05cf-4bcc-83d3-a7d51ffc9977"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:15:42.087688 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.087661    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x"]
Oct 30 19:15:42.105882 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.105856    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-314bd6fa-rs24x"]
Oct 30 19:15:42.129186 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.129157    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:42.129186 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.129180    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:42.129328 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.129197    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:42.129328 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.129212    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88w66\" (UniqueName: \"kubernetes.io/projected/c01a04de-05cf-4bcc-83d3-a7d51ffc9977-kube-api-access-88w66\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:42.163050 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.162969    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:42.416915 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.416827    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"]
Oct 30 19:15:42.423632 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.423602    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-568107-openshift-default-76c65fb5d5-fdkcg"]
Oct 30 19:15:42.531816 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.531780    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:42.837096 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.837000    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04a304ec-392b-4c28-bb4a-f20d79e1dbee" path="/var/lib/kubelet/pods/04a304ec-392b-4c28-bb4a-f20d79e1dbee/volumes"
Oct 30 19:15:42.837521 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:42.837505    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c01a04de-05cf-4bcc-83d3-a7d51ffc9977" path="/var/lib/kubelet/pods/c01a04de-05cf-4bcc-83d3-a7d51ffc9977/volumes"
Oct 30 19:15:43.184939 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:43.184906    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n"
Oct 30 19:15:43.194041 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:43.193995    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:43.344052 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:43.344017    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cab88e32-0265-482d-b3f1-a2e33f1dd3ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cab88e32-0265-482d-b3f1-a2e33f1dd3ac-maas-api-openshift-ca\") pod \"cab88e32-0265-482d-b3f1-a2e33f1dd3ac\" (UID: \"cab88e32-0265-482d-b3f1-a2e33f1dd3ac\") "
Oct 30 19:15:43.344274 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:43.344173    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab88e32-0265-482d-b3f1-a2e33f1dd3ac-kube-api-access-dj7zf\" (UniqueName: \"kubernetes.io/projected/cab88e32-0265-482d-b3f1-a2e33f1dd3ac-kube-api-access-dj7zf\") pod \"cab88e32-0265-482d-b3f1-a2e33f1dd3ac\" (UID: \"cab88e32-0265-482d-b3f1-a2e33f1dd3ac\") "
Oct 30 19:15:43.344274 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:43.344209    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cab88e32-0265-482d-b3f1-a2e33f1dd3ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cab88e32-0265-482d-b3f1-a2e33f1dd3ac-maas-api-xks-ca\") pod \"cab88e32-0265-482d-b3f1-a2e33f1dd3ac\" (UID: \"cab88e32-0265-482d-b3f1-a2e33f1dd3ac\") "
Oct 30 19:15:43.344473 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:43.344446    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab88e32-0265-482d-b3f1-a2e33f1dd3ac-maas-api-openshift-ca" pod "cab88e32-0265-482d-b3f1-a2e33f1dd3ac" (UID: "cab88e32-0265-482d-b3f1-a2e33f1dd3ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:43.344769 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:43.344749    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cab88e32-0265-482d-b3f1-a2e33f1dd3ac-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:43.346480 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:43.346451    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cab88e32-0265-482d-b3f1-a2e33f1dd3ac-maas-api-xks-ca" pod "cab88e32-0265-482d-b3f1-a2e33f1dd3ac" (UID: "cab88e32-0265-482d-b3f1-a2e33f1dd3ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:43.346572 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:43.346532    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab88e32-0265-482d-b3f1-a2e33f1dd3ac-kube-api-access-dj7zf" pod "cab88e32-0265-482d-b3f1-a2e33f1dd3ac" (UID: "cab88e32-0265-482d-b3f1-a2e33f1dd3ac"). InnerVolumeSpecName "kube-api-access-dj7zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:43.446580 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:43.446501    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj7zf\" (UniqueName: \"kubernetes.io/projected/cab88e32-0265-482d-b3f1-a2e33f1dd3ac-kube-api-access-dj7zf\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:43.446580 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:43.446534    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cab88e32-0265-482d-b3f1-a2e33f1dd3ac-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:43.503769 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:43.503736    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:43.684096 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:43.683238    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"]
Oct 30 19:15:43.690646 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:43.690613    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"]
Oct 30 19:15:43.805112 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:43.805007    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:44.016349 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.016319    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n"
Oct 30 19:15:44.016514 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.016375    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n" event={"ID":"cab88e32-0265-482d-b3f1-a2e33f1dd3ac","Type":"ContainerDied","Data":"86e1808c11caf78241caec246396974764f2634abbbe7f9a10d60fda2680b9e3"}
Oct 30 19:15:44.016514 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.016425    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86e1808c11caf78241caec246396974764f2634abbbe7f9a10d60fda2680b9e3"
Oct 30 19:15:44.067977 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.067898    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:44.120514 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.120479    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh"]
Oct 30 19:15:44.121775 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.121725    4924 cpu_manager.go:414] "containerMap: removing container" podUID="c01a04de-05cf-4bcc-83d3-a7d51ffc9977" containerName="istio-proxy"
Oct 30 19:15:44.121775 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.121748    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c01a04de-05cf-4bcc-83d3-a7d51ffc9977" containerName="istio-proxy"
Oct 30 19:15:44.121981 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.121801    4924 cpu_manager.go:414] "containerMap: removing container" podUID="c01a04de-05cf-4bcc-83d3-a7d51ffc9977" containerName="istio-proxy"
Oct 30 19:15:44.121981 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.121811    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c01a04de-05cf-4bcc-83d3-a7d51ffc9977" containerName="istio-proxy"
Oct 30 19:15:44.121981 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.121864    4924 cpu_manager.go:414] "containerMap: removing container" podUID="cab88e32-0265-482d-b3f1-a2e33f1dd3ac" containerName="revoke-keys"
Oct 30 19:15:44.121981 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.121873    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab88e32-0265-482d-b3f1-a2e33f1dd3ac" containerName="revoke-keys"
Oct 30 19:15:44.122235 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.122039    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab88e32-0265-482d-b3f1-a2e33f1dd3ac" containerName="revoke-keys"
Oct 30 19:15:44.122235 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.122107    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="c01a04de-05cf-4bcc-83d3-a7d51ffc9977" containerName="istio-proxy"
Oct 30 19:15:44.122235 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.122129    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="c01a04de-05cf-4bcc-83d3-a7d51ffc9977" containerName="istio-proxy"
Oct 30 19:15:44.125940 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.125921    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh"
Oct 30 19:15:44.178881 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.178837    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:15:44.179223 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.179200    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:15:44.179223 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.179226    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:15:44.179401 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.179250    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:15:44.183606 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.183586    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh"]
Oct 30 19:15:44.257076 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.257033    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1a3b34a-c6c5-483a-a419-784a3e5c95a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh\" (UID: \"e1a3b34a-c6c5-483a-a419-784a3e5c95a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh"
Oct 30 19:15:44.257532 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.257159    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5c5b\" (UniqueName: \"kubernetes.io/projected/e1a3b34a-c6c5-483a-a419-784a3e5c95a5-kube-api-access-r5c5b\") pod \"maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh\" (UID: \"e1a3b34a-c6c5-483a-a419-784a3e5c95a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh"
Oct 30 19:15:44.257532 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.257369    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1a3b34a-c6c5-483a-a419-784a3e5c95a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh\" (UID: \"e1a3b34a-c6c5-483a-a419-784a3e5c95a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh"
Oct 30 19:15:44.358561 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.358528    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1a3b34a-c6c5-483a-a419-784a3e5c95a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh\" (UID: \"e1a3b34a-c6c5-483a-a419-784a3e5c95a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh"
Oct 30 19:15:44.358724 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.358641    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1a3b34a-c6c5-483a-a419-784a3e5c95a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh\" (UID: \"e1a3b34a-c6c5-483a-a419-784a3e5c95a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh"
Oct 30 19:15:44.358724 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.358688    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5c5b\" (UniqueName: \"kubernetes.io/projected/e1a3b34a-c6c5-483a-a419-784a3e5c95a5-kube-api-access-r5c5b\") pod \"maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh\" (UID: \"e1a3b34a-c6c5-483a-a419-784a3e5c95a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh"
Oct 30 19:15:44.359273 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.359253    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1a3b34a-c6c5-483a-a419-784a3e5c95a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh\" (UID: \"e1a3b34a-c6c5-483a-a419-784a3e5c95a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh"
Oct 30 19:15:44.361006 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.360987    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1a3b34a-c6c5-483a-a419-784a3e5c95a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh\" (UID: \"e1a3b34a-c6c5-483a-a419-784a3e5c95a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh"
Oct 30 19:15:44.420655 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.420626    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5c5b\" (UniqueName: \"kubernetes.io/projected/e1a3b34a-c6c5-483a-a419-784a3e5c95a5-kube-api-access-r5c5b\") pod \"maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh\" (UID: \"e1a3b34a-c6c5-483a-a419-784a3e5c95a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh"
Oct 30 19:15:44.442494 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.442462    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh"
Oct 30 19:15:44.821426 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:44.821388    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh"]
Oct 30 19:15:44.821856 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:15:44.821820    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a3b34a_c6c5_483a_a419_784a3e5c95a5.slice/crio-81d8764abfebd6be81c4edd387aace33a98317d429fcea0ada1813eba28a9a91 WatchSource:0}: Error finding container 81d8764abfebd6be81c4edd387aace33a98317d429fcea0ada1813eba28a9a91: Status 404 returned error can't find the container with id 81d8764abfebd6be81c4edd387aace33a98317d429fcea0ada1813eba28a9a91
Oct 30 19:15:45.024817 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:45.024785    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh" event={"ID":"e1a3b34a-c6c5-483a-a419-784a3e5c95a5","Type":"ContainerStarted","Data":"81d8764abfebd6be81c4edd387aace33a98317d429fcea0ada1813eba28a9a91"}
Oct 30 19:15:45.713028 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:45.712993    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:15:46.033553 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:46.033472    4924 generic.go:356] "Generic (PLEG): container finished" podID="e1a3b34a-c6c5-483a-a419-784a3e5c95a5" containerID="88a3779d8e6f0f7d5b5d953e8c6570dcb992e25b37bd8ad367bffd023eef3e62" exitCode=0
Oct 30 19:15:46.033553 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:46.033518    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh" event={"ID":"e1a3b34a-c6c5-483a-a419-784a3e5c95a5","Type":"ContainerDied","Data":"88a3779d8e6f0f7d5b5d953e8c6570dcb992e25b37bd8ad367bffd023eef3e62"}
Oct 30 19:15:47.190520 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:47.190495    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh"
Oct 30 19:15:47.232166 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:47.230975    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:47.299532 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:47.299448    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1a3b34a-c6c5-483a-a419-784a3e5c95a5-kube-api-access-r5c5b\" (UniqueName: \"kubernetes.io/projected/e1a3b34a-c6c5-483a-a419-784a3e5c95a5-kube-api-access-r5c5b\") pod \"e1a3b34a-c6c5-483a-a419-784a3e5c95a5\" (UID: \"e1a3b34a-c6c5-483a-a419-784a3e5c95a5\") "
Oct 30 19:15:47.299532 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:47.299514    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1a3b34a-c6c5-483a-a419-784a3e5c95a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1a3b34a-c6c5-483a-a419-784a3e5c95a5-maas-api-xks-ca\") pod \"e1a3b34a-c6c5-483a-a419-784a3e5c95a5\" (UID: \"e1a3b34a-c6c5-483a-a419-784a3e5c95a5\") "
Oct 30 19:15:47.299944 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:47.299756    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1a3b34a-c6c5-483a-a419-784a3e5c95a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1a3b34a-c6c5-483a-a419-784a3e5c95a5-maas-api-openshift-ca\") pod \"e1a3b34a-c6c5-483a-a419-784a3e5c95a5\" (UID: \"e1a3b34a-c6c5-483a-a419-784a3e5c95a5\") "
Oct 30 19:15:47.300383 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:47.300352    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1a3b34a-c6c5-483a-a419-784a3e5c95a5-maas-api-openshift-ca" pod "e1a3b34a-c6c5-483a-a419-784a3e5c95a5" (UID: "e1a3b34a-c6c5-483a-a419-784a3e5c95a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:47.302519 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:47.302493    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1a3b34a-c6c5-483a-a419-784a3e5c95a5-maas-api-xks-ca" pod "e1a3b34a-c6c5-483a-a419-784a3e5c95a5" (UID: "e1a3b34a-c6c5-483a-a419-784a3e5c95a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:47.302519 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:47.302509    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a3b34a-c6c5-483a-a419-784a3e5c95a5-kube-api-access-r5c5b" pod "e1a3b34a-c6c5-483a-a419-784a3e5c95a5" (UID: "e1a3b34a-c6c5-483a-a419-784a3e5c95a5"). InnerVolumeSpecName "kube-api-access-r5c5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:47.401287 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:47.401252    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1a3b34a-c6c5-483a-a419-784a3e5c95a5-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:47.401287 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:47.401281    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5c5b\" (UniqueName: \"kubernetes.io/projected/e1a3b34a-c6c5-483a-a419-784a3e5c95a5-kube-api-access-r5c5b\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:47.401287 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:47.401292    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1a3b34a-c6c5-483a-a419-784a3e5c95a5-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:47.575565 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:47.575475    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:48.058094 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:48.056647    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh" event={"ID":"e1a3b34a-c6c5-483a-a419-784a3e5c95a5","Type":"ContainerDied","Data":"81d8764abfebd6be81c4edd387aace33a98317d429fcea0ada1813eba28a9a91"}
Oct 30 19:15:48.058094 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:48.056713    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81d8764abfebd6be81c4edd387aace33a98317d429fcea0ada1813eba28a9a91"
Oct 30 19:15:48.058094 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:48.056789    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh"
Oct 30 19:15:48.124087 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:48.120401    4924 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a3b34a_c6c5_483a_a419_784a3e5c95a5.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 19:15:48.404493 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:48.404454    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6" podUID="e9720622-6440-4232-8855-ae46ec5bf434" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:15:48.405046 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:48.405018    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb" podUID="9918ebe6-aa19-488e-b259-e6201f326650" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:15:50.670661 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:50.670627    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs"]
Oct 30 19:15:50.831339 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:50.831311    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6"
Oct 30 19:15:50.849177 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:50.849149    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9720622-6440-4232-8855-ae46ec5bf434-kube-api-access-g67jv\" (UniqueName: \"kubernetes.io/projected/e9720622-6440-4232-8855-ae46ec5bf434-kube-api-access-g67jv\") pod \"e9720622-6440-4232-8855-ae46ec5bf434\" (UID: \"e9720622-6440-4232-8855-ae46ec5bf434\") "
Oct 30 19:15:50.849295 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:50.849270    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9720622-6440-4232-8855-ae46ec5bf434-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9720622-6440-4232-8855-ae46ec5bf434-plugins-config-volume\") pod \"e9720622-6440-4232-8855-ae46ec5bf434\" (UID: \"e9720622-6440-4232-8855-ae46ec5bf434\") "
Oct 30 19:15:50.849633 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:50.849612    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9720622-6440-4232-8855-ae46ec5bf434-plugins-config-volume" pod "e9720622-6440-4232-8855-ae46ec5bf434" (UID: "e9720622-6440-4232-8855-ae46ec5bf434"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:50.850098 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:50.850075    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9720622-6440-4232-8855-ae46ec5bf434-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:50.851367 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:50.851348    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9720622-6440-4232-8855-ae46ec5bf434-kube-api-access-g67jv" pod "e9720622-6440-4232-8855-ae46ec5bf434" (UID: "e9720622-6440-4232-8855-ae46ec5bf434"). InnerVolumeSpecName "kube-api-access-g67jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:50.862566 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:50.862520    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb"
Oct 30 19:15:50.951823 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:50.951461    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9918ebe6-aa19-488e-b259-e6201f326650-kube-api-access-w54r8\" (UniqueName: \"kubernetes.io/projected/9918ebe6-aa19-488e-b259-e6201f326650-kube-api-access-w54r8\") pod \"9918ebe6-aa19-488e-b259-e6201f326650\" (UID: \"9918ebe6-aa19-488e-b259-e6201f326650\") "
Oct 30 19:15:50.951823 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:50.951691    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9918ebe6-aa19-488e-b259-e6201f326650-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9918ebe6-aa19-488e-b259-e6201f326650-plugins-config-volume\") pod \"9918ebe6-aa19-488e-b259-e6201f326650\" (UID: \"9918ebe6-aa19-488e-b259-e6201f326650\") "
Oct 30 19:15:50.952203 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:50.952181    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g67jv\" (UniqueName: \"kubernetes.io/projected/e9720622-6440-4232-8855-ae46ec5bf434-kube-api-access-g67jv\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:50.953001 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:50.952972    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9918ebe6-aa19-488e-b259-e6201f326650-plugins-config-volume" pod "9918ebe6-aa19-488e-b259-e6201f326650" (UID: "9918ebe6-aa19-488e-b259-e6201f326650"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:50.960256 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:50.960226    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9918ebe6-aa19-488e-b259-e6201f326650-kube-api-access-w54r8" pod "9918ebe6-aa19-488e-b259-e6201f326650" (UID: "9918ebe6-aa19-488e-b259-e6201f326650"). InnerVolumeSpecName "kube-api-access-w54r8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:51.053654 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:51.053577    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9918ebe6-aa19-488e-b259-e6201f326650-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:51.053654 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:51.053606    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w54r8\" (UniqueName: \"kubernetes.io/projected/9918ebe6-aa19-488e-b259-e6201f326650-kube-api-access-w54r8\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:51.062250 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:51.062225    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:51.082797 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:51.082761    4924 generic.go:356] "Generic (PLEG): container finished" podID="e9720622-6440-4232-8855-ae46ec5bf434" containerID="a5aefc16cb3c8a3ec160196d15b320d3c3540f6ea9f9e35e6992355eb605d3ab" exitCode=0
Oct 30 19:15:51.083014 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:51.082847    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6"
Oct 30 19:15:51.083014 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:51.082920    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6" event={"ID":"e9720622-6440-4232-8855-ae46ec5bf434","Type":"ContainerDied","Data":"a5aefc16cb3c8a3ec160196d15b320d3c3540f6ea9f9e35e6992355eb605d3ab"}
Oct 30 19:15:51.083014 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:51.083000    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6" event={"ID":"e9720622-6440-4232-8855-ae46ec5bf434","Type":"ContainerDied","Data":"ecd2b8fa8dfa045f71bbb37f1141717bcaaff30fd588e442b4f44cbf0fe1cc2d"}
Oct 30 19:15:51.083277 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:51.083029    4924 scope.go:122] "RemoveContainer" containerID="a5aefc16cb3c8a3ec160196d15b320d3c3540f6ea9f9e35e6992355eb605d3ab"
Oct 30 19:15:51.087480 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:51.087458    4924 generic.go:356] "Generic (PLEG): container finished" podID="9918ebe6-aa19-488e-b259-e6201f326650" containerID="7ecc1ae5bfa9ec792e152a062207ab7d74b9b753618c3c9f21d203482b768de3" exitCode=0
Oct 30 19:15:51.087772 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:51.087751    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb" event={"ID":"9918ebe6-aa19-488e-b259-e6201f326650","Type":"ContainerDied","Data":"7ecc1ae5bfa9ec792e152a062207ab7d74b9b753618c3c9f21d203482b768de3"}
Oct 30 19:15:51.087920 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:51.087902    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb" event={"ID":"9918ebe6-aa19-488e-b259-e6201f326650","Type":"ContainerDied","Data":"5a1f856a76ff4cb277d6e0eff14159660b38996332f1ff9e63521ed7fb2eeeed"}
Oct 30 19:15:51.088104 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:51.088087    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb"
Oct 30 19:15:51.099646 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:51.099625    4924 scope.go:122] "RemoveContainer" containerID="a5aefc16cb3c8a3ec160196d15b320d3c3540f6ea9f9e35e6992355eb605d3ab"
Oct 30 19:15:51.100279 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:51.099932    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5aefc16cb3c8a3ec160196d15b320d3c3540f6ea9f9e35e6992355eb605d3ab\": container with ID starting with a5aefc16cb3c8a3ec160196d15b320d3c3540f6ea9f9e35e6992355eb605d3ab not found: ID does not exist" containerID="a5aefc16cb3c8a3ec160196d15b320d3c3540f6ea9f9e35e6992355eb605d3ab"
Oct 30 19:15:51.100279 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:51.099965    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5aefc16cb3c8a3ec160196d15b320d3c3540f6ea9f9e35e6992355eb605d3ab"} err="failed to get container status \"a5aefc16cb3c8a3ec160196d15b320d3c3540f6ea9f9e35e6992355eb605d3ab\": rpc error: code = NotFound desc = could not find container \"a5aefc16cb3c8a3ec160196d15b320d3c3540f6ea9f9e35e6992355eb605d3ab\": container with ID starting with a5aefc16cb3c8a3ec160196d15b320d3c3540f6ea9f9e35e6992355eb605d3ab not found: ID does not exist"
Oct 30 19:15:51.100279 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:51.099989    4924 scope.go:122] "RemoveContainer" containerID="7ecc1ae5bfa9ec792e152a062207ab7d74b9b753618c3c9f21d203482b768de3"
Oct 30 19:15:51.113800 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:51.113782    4924 scope.go:122] "RemoveContainer" containerID="7ecc1ae5bfa9ec792e152a062207ab7d74b9b753618c3c9f21d203482b768de3"
Oct 30 19:15:51.114119 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:51.114100    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ecc1ae5bfa9ec792e152a062207ab7d74b9b753618c3c9f21d203482b768de3\": container with ID starting with 7ecc1ae5bfa9ec792e152a062207ab7d74b9b753618c3c9f21d203482b768de3 not found: ID does not exist" containerID="7ecc1ae5bfa9ec792e152a062207ab7d74b9b753618c3c9f21d203482b768de3"
Oct 30 19:15:51.114186 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:51.114126    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ecc1ae5bfa9ec792e152a062207ab7d74b9b753618c3c9f21d203482b768de3"} err="failed to get container status \"7ecc1ae5bfa9ec792e152a062207ab7d74b9b753618c3c9f21d203482b768de3\": rpc error: code = NotFound desc = could not find container \"7ecc1ae5bfa9ec792e152a062207ab7d74b9b753618c3c9f21d203482b768de3\": container with ID starting with 7ecc1ae5bfa9ec792e152a062207ab7d74b9b753618c3c9f21d203482b768de3 not found: ID does not exist"
Oct 30 19:15:51.241873 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:51.241834    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb"]
Oct 30 19:15:51.255424 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:51.255393    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f80ac-8696db7479-ktcmb"]
Oct 30 19:15:51.259706 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:51.259675    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:51.360463 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:51.360435    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6"]
Oct 30 19:15:51.368658 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:51.368628    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5f80ac-7c54fd6c7d-zn4s6"]
Oct 30 19:15:52.466186 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:52.464600    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:52.835143 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:52.835031    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9918ebe6-aa19-488e-b259-e6201f326650" path="/var/lib/kubelet/pods/9918ebe6-aa19-488e-b259-e6201f326650/volumes"
Oct 30 19:15:52.835517 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:52.835494    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9720622-6440-4232-8855-ae46ec5bf434" path="/var/lib/kubelet/pods/e9720622-6440-4232-8855-ae46ec5bf434/volumes"
Oct 30 19:15:54.943205 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:54.943166    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:55.153091 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:55.153028    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:55.580940 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:55.580900    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f"]
Oct 30 19:15:55.671351 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:55.671297    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs" podUID="156d3474-d19a-4022-9a3c-88b93e53b527" containerName="payload-processing" containerID="cri-o://a738ae0497f8e4bd70d0d7068ee0873ba6e2c610f42eb86962fd63ef69bbbbdc" gracePeriod=25
Oct 30 19:15:55.683679 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:55.683653    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x"]
Oct 30 19:15:55.994030 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:55.993994    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:56.205985 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:56.205952    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:56.585239 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:56.585196    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:57.369266 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:57.369217    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:15:58.720044 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:58.720004    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt" podUID="955e98b9-864b-4636-9511-8b69f981eaa7" containerName="main" containerID="cri-o://38c70d190e6fc191ab9f19230eb117760419769282ceed555089a66a0610698d" gracePeriod=45
Oct 30 19:15:58.721722 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:58.721691    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l" podUID="bdc8df4b-8c94-40b8-b4be-80d1432d87de" containerName="main" containerID="cri-o://1f766806088d6f7a30f731ecb880a0a63113e8f4b3b7e11b044b92f5e4a399d8" gracePeriod=45
Oct 30 19:15:59.047787 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.047761    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 19:15:59.050162 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.050140    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 19:15:59.166009 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.165940    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdc8df4b-8c94-40b8-b4be-80d1432d87de-kube-api-access-4czhc\" (UniqueName: \"kubernetes.io/projected/bdc8df4b-8c94-40b8-b4be-80d1432d87de-kube-api-access-4czhc\") pod \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\" (UID: \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\") "
Oct 30 19:15:59.170619 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.170597    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-home\" (UniqueName: \"kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-home\") pod \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\" (UID: \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\") "
Oct 30 19:15:59.170876 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.170860    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-dshm\") pod \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\" (UID: \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\") "
Oct 30 19:15:59.171116 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.171100    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/955e98b9-864b-4636-9511-8b69f981eaa7-tls-certs\" (UniqueName: \"kubernetes.io/secret/955e98b9-864b-4636-9511-8b69f981eaa7-tls-certs\") pod \"955e98b9-864b-4636-9511-8b69f981eaa7\" (UID: \"955e98b9-864b-4636-9511-8b69f981eaa7\") "
Oct 30 19:15:59.171256 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.171242    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-tmp-dir\") pod \"955e98b9-864b-4636-9511-8b69f981eaa7\" (UID: \"955e98b9-864b-4636-9511-8b69f981eaa7\") "
Oct 30 19:15:59.171381 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.171364    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-home\" (UniqueName: \"kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-home\") pod \"955e98b9-864b-4636-9511-8b69f981eaa7\" (UID: \"955e98b9-864b-4636-9511-8b69f981eaa7\") "
Oct 30 19:15:59.171494 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.171472    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-dshm\") pod \"955e98b9-864b-4636-9511-8b69f981eaa7\" (UID: \"955e98b9-864b-4636-9511-8b69f981eaa7\") "
Oct 30 19:15:59.171602 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.171587    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-tmp-dir\") pod \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\" (UID: \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\") "
Oct 30 19:15:59.171681 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.171634    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdc8df4b-8c94-40b8-b4be-80d1432d87de-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdc8df4b-8c94-40b8-b4be-80d1432d87de-tls-certs\") pod \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\" (UID: \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\") "
Oct 30 19:15:59.171776 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.171761    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-model-cache\") pod \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\" (UID: \"bdc8df4b-8c94-40b8-b4be-80d1432d87de\") "
Oct 30 19:15:59.171837 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.171793    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-model-cache\") pod \"955e98b9-864b-4636-9511-8b69f981eaa7\" (UID: \"955e98b9-864b-4636-9511-8b69f981eaa7\") "
Oct 30 19:15:59.171837 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.171824    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/955e98b9-864b-4636-9511-8b69f981eaa7-kube-api-access-dldgw\" (UniqueName: \"kubernetes.io/projected/955e98b9-864b-4636-9511-8b69f981eaa7-kube-api-access-dldgw\") pod \"955e98b9-864b-4636-9511-8b69f981eaa7\" (UID: \"955e98b9-864b-4636-9511-8b69f981eaa7\") "
Oct 30 19:15:59.172136 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.172107    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc8df4b-8c94-40b8-b4be-80d1432d87de-kube-api-access-4czhc" pod "bdc8df4b-8c94-40b8-b4be-80d1432d87de" (UID: "bdc8df4b-8c94-40b8-b4be-80d1432d87de"). InnerVolumeSpecName "kube-api-access-4czhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:59.172422 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.172401    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4czhc\" (UniqueName: \"kubernetes.io/projected/bdc8df4b-8c94-40b8-b4be-80d1432d87de-kube-api-access-4czhc\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:59.172778 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.172753    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-tmp-dir" pod "955e98b9-864b-4636-9511-8b69f981eaa7" (UID: "955e98b9-864b-4636-9511-8b69f981eaa7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:59.172907 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.168480    4924 generic.go:356] "Generic (PLEG): container finished" podID="955e98b9-864b-4636-9511-8b69f981eaa7" containerID="38c70d190e6fc191ab9f19230eb117760419769282ceed555089a66a0610698d" exitCode=0
Oct 30 19:15:59.174257 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.174230    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-model-cache" pod "bdc8df4b-8c94-40b8-b4be-80d1432d87de" (UID: "bdc8df4b-8c94-40b8-b4be-80d1432d87de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:59.174389 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.174368    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-home" pod "bdc8df4b-8c94-40b8-b4be-80d1432d87de" (UID: "bdc8df4b-8c94-40b8-b4be-80d1432d87de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:59.174474 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.168597    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"
Oct 30 19:15:59.176126 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.175836    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-model-cache" pod "955e98b9-864b-4636-9511-8b69f981eaa7" (UID: "955e98b9-864b-4636-9511-8b69f981eaa7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:59.176126 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.168512    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt" event={"ID":"955e98b9-864b-4636-9511-8b69f981eaa7","Type":"ContainerDied","Data":"38c70d190e6fc191ab9f19230eb117760419769282ceed555089a66a0610698d"}
Oct 30 19:15:59.176126 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.175906    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt" event={"ID":"955e98b9-864b-4636-9511-8b69f981eaa7","Type":"ContainerDied","Data":"a80db07cbf793cdcc54b890292c9297d9f7964d55655a5dbde6515ede05583aa"}
Oct 30 19:15:59.176126 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.175938    4924 scope.go:122] "RemoveContainer" containerID="38c70d190e6fc191ab9f19230eb117760419769282ceed555089a66a0610698d"
Oct 30 19:15:59.176398 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.176243    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-home" pod "955e98b9-864b-4636-9511-8b69f981eaa7" (UID: "955e98b9-864b-4636-9511-8b69f981eaa7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:59.176642 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.176618    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-tmp-dir" pod "bdc8df4b-8c94-40b8-b4be-80d1432d87de" (UID: "bdc8df4b-8c94-40b8-b4be-80d1432d87de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:59.178911 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.178887    4924 generic.go:356] "Generic (PLEG): container finished" podID="bdc8df4b-8c94-40b8-b4be-80d1432d87de" containerID="1f766806088d6f7a30f731ecb880a0a63113e8f4b3b7e11b044b92f5e4a399d8" exitCode=0
Oct 30 19:15:59.179238 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.178957    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"
Oct 30 19:15:59.179347 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.178973    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l" event={"ID":"bdc8df4b-8c94-40b8-b4be-80d1432d87de","Type":"ContainerDied","Data":"1f766806088d6f7a30f731ecb880a0a63113e8f4b3b7e11b044b92f5e4a399d8"}
Oct 30 19:15:59.179347 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.179340    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l" event={"ID":"bdc8df4b-8c94-40b8-b4be-80d1432d87de","Type":"ContainerDied","Data":"b70973ac682e2ac4c555b24b7a01855049f7ccf4612cbe6b9d37a50d2c26872a"}
Oct 30 19:15:59.179473 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.178979    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/955e98b9-864b-4636-9511-8b69f981eaa7-tls-certs" pod "955e98b9-864b-4636-9511-8b69f981eaa7" (UID: "955e98b9-864b-4636-9511-8b69f981eaa7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:59.188410 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.188380    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-dshm" pod "bdc8df4b-8c94-40b8-b4be-80d1432d87de" (UID: "bdc8df4b-8c94-40b8-b4be-80d1432d87de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:59.188539 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.188422    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdc8df4b-8c94-40b8-b4be-80d1432d87de-tls-certs" pod "bdc8df4b-8c94-40b8-b4be-80d1432d87de" (UID: "bdc8df4b-8c94-40b8-b4be-80d1432d87de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:59.188539 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.188467    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955e98b9-864b-4636-9511-8b69f981eaa7-kube-api-access-dldgw" pod "955e98b9-864b-4636-9511-8b69f981eaa7" (UID: "955e98b9-864b-4636-9511-8b69f981eaa7"). InnerVolumeSpecName "kube-api-access-dldgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:59.189742 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.189716    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-dshm" pod "955e98b9-864b-4636-9511-8b69f981eaa7" (UID: "955e98b9-864b-4636-9511-8b69f981eaa7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:59.191031 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.191008    4924 scope.go:122] "RemoveContainer" containerID="38c70d190e6fc191ab9f19230eb117760419769282ceed555089a66a0610698d"
Oct 30 19:15:59.191448 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:59.191425    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38c70d190e6fc191ab9f19230eb117760419769282ceed555089a66a0610698d\": container with ID starting with 38c70d190e6fc191ab9f19230eb117760419769282ceed555089a66a0610698d not found: ID does not exist" containerID="38c70d190e6fc191ab9f19230eb117760419769282ceed555089a66a0610698d"
Oct 30 19:15:59.191544 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.191459    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38c70d190e6fc191ab9f19230eb117760419769282ceed555089a66a0610698d"} err="failed to get container status \"38c70d190e6fc191ab9f19230eb117760419769282ceed555089a66a0610698d\": rpc error: code = NotFound desc = could not find container \"38c70d190e6fc191ab9f19230eb117760419769282ceed555089a66a0610698d\": container with ID starting with 38c70d190e6fc191ab9f19230eb117760419769282ceed555089a66a0610698d not found: ID does not exist"
Oct 30 19:15:59.191544 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.191483    4924 scope.go:122] "RemoveContainer" containerID="1f766806088d6f7a30f731ecb880a0a63113e8f4b3b7e11b044b92f5e4a399d8"
Oct 30 19:15:59.201353 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.201328    4924 scope.go:122] "RemoveContainer" containerID="1f766806088d6f7a30f731ecb880a0a63113e8f4b3b7e11b044b92f5e4a399d8"
Oct 30 19:15:59.201653 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:15:59.201628    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f766806088d6f7a30f731ecb880a0a63113e8f4b3b7e11b044b92f5e4a399d8\": container with ID starting with 1f766806088d6f7a30f731ecb880a0a63113e8f4b3b7e11b044b92f5e4a399d8 not found: ID does not exist" containerID="1f766806088d6f7a30f731ecb880a0a63113e8f4b3b7e11b044b92f5e4a399d8"
Oct 30 19:15:59.201753 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.201663    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f766806088d6f7a30f731ecb880a0a63113e8f4b3b7e11b044b92f5e4a399d8"} err="failed to get container status \"1f766806088d6f7a30f731ecb880a0a63113e8f4b3b7e11b044b92f5e4a399d8\": rpc error: code = NotFound desc = could not find container \"1f766806088d6f7a30f731ecb880a0a63113e8f4b3b7e11b044b92f5e4a399d8\": container with ID starting with 1f766806088d6f7a30f731ecb880a0a63113e8f4b3b7e11b044b92f5e4a399d8 not found: ID does not exist"
Oct 30 19:15:59.276233 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.275876    4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-tmp-dir\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:59.276233 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.275908    4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdc8df4b-8c94-40b8-b4be-80d1432d87de-tls-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:59.276233 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.275923    4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-model-cache\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:59.276233 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.275942    4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-model-cache\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:59.276233 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.275957    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dldgw\" (UniqueName: \"kubernetes.io/projected/955e98b9-864b-4636-9511-8b69f981eaa7-kube-api-access-dldgw\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:59.276233 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.275971    4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-home\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:59.276233 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.275985    4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdc8df4b-8c94-40b8-b4be-80d1432d87de-dshm\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:59.276233 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.275998    4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955e98b9-864b-4636-9511-8b69f981eaa7-tls-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:59.276233 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.276010    4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-tmp-dir\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:59.276233 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.276018    4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-home\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:59.276233 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.276026    4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955e98b9-864b-4636-9511-8b69f981eaa7-dshm\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:59.591563 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.591487    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"]
Oct 30 19:15:59.597525 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.597499    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a1617e/distinct-2-w3-a1617e-kserve-6c6668ddb-6gxwt"]
Oct 30 19:15:59.705374 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.705340    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"]
Oct 30 19:15:59.710046 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.710023    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l"]
Oct 30 19:15:59.977852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:15:59.977815    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a1617e/distinct-w3-a1617e-kserve-6fdbdb49d9-mvt2l" podUID="bdc8df4b-8c94-40b8-b4be-80d1432d87de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:16:00.582217 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:00.582172    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f" podUID="9326b21d-3f94-4d9c-99f5-93e96de020bf" containerName="payload-processing" containerID="cri-o://aa6bdf60801d333c730aeedce620826bb0b842d2457a5830cab43b4aaa74970f" gracePeriod=25
Oct 30 19:16:00.684517 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:00.684478    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x" podUID="0e8d5d76-7d10-4af1-8689-77dd76070856" containerName="payload-pre-processing" containerID="cri-o://2be0add783dbbcab1079ffc5bf984b5619a3ebb21434e38a630b44c55682e773" gracePeriod=25
Oct 30 19:16:00.840723 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:00.840216    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="955e98b9-864b-4636-9511-8b69f981eaa7" path="/var/lib/kubelet/pods/955e98b9-864b-4636-9511-8b69f981eaa7/volumes"
Oct 30 19:16:00.840915 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:00.840893    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdc8df4b-8c94-40b8-b4be-80d1432d87de" path="/var/lib/kubelet/pods/bdc8df4b-8c94-40b8-b4be-80d1432d87de/volumes"
Oct 30 19:16:00.860923 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:00.860894    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f"
Oct 30 19:16:00.997899 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:00.997871    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x"
Oct 30 19:16:01.001522 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.001495    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9326b21d-3f94-4d9c-99f5-93e96de020bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9326b21d-3f94-4d9c-99f5-93e96de020bf-plugins-config-volume\") pod \"9326b21d-3f94-4d9c-99f5-93e96de020bf\" (UID: \"9326b21d-3f94-4d9c-99f5-93e96de020bf\") "
Oct 30 19:16:01.001651 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.001635    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9326b21d-3f94-4d9c-99f5-93e96de020bf-kube-api-access-jw6xg\" (UniqueName: \"kubernetes.io/projected/9326b21d-3f94-4d9c-99f5-93e96de020bf-kube-api-access-jw6xg\") pod \"9326b21d-3f94-4d9c-99f5-93e96de020bf\" (UID: \"9326b21d-3f94-4d9c-99f5-93e96de020bf\") "
Oct 30 19:16:01.001938 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.001908    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9326b21d-3f94-4d9c-99f5-93e96de020bf-plugins-config-volume" pod "9326b21d-3f94-4d9c-99f5-93e96de020bf" (UID: "9326b21d-3f94-4d9c-99f5-93e96de020bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:01.002249 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.002221    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9326b21d-3f94-4d9c-99f5-93e96de020bf-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.003726 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.003699    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9326b21d-3f94-4d9c-99f5-93e96de020bf-kube-api-access-jw6xg" pod "9326b21d-3f94-4d9c-99f5-93e96de020bf" (UID: "9326b21d-3f94-4d9c-99f5-93e96de020bf"). InnerVolumeSpecName "kube-api-access-jw6xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:01.103285 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.103259    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e8d5d76-7d10-4af1-8689-77dd76070856-kube-api-access-v6n9q\" (UniqueName: \"kubernetes.io/projected/0e8d5d76-7d10-4af1-8689-77dd76070856-kube-api-access-v6n9q\") pod \"0e8d5d76-7d10-4af1-8689-77dd76070856\" (UID: \"0e8d5d76-7d10-4af1-8689-77dd76070856\") "
Oct 30 19:16:01.103438 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.103296    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e8d5d76-7d10-4af1-8689-77dd76070856-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8d5d76-7d10-4af1-8689-77dd76070856-plugins-config-volume\") pod \"0e8d5d76-7d10-4af1-8689-77dd76070856\" (UID: \"0e8d5d76-7d10-4af1-8689-77dd76070856\") "
Oct 30 19:16:01.103702 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.103676    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e8d5d76-7d10-4af1-8689-77dd76070856-plugins-config-volume" pod "0e8d5d76-7d10-4af1-8689-77dd76070856" (UID: "0e8d5d76-7d10-4af1-8689-77dd76070856"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:01.103702 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.103684    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw6xg\" (UniqueName: \"kubernetes.io/projected/9326b21d-3f94-4d9c-99f5-93e96de020bf-kube-api-access-jw6xg\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.105455 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.105431    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8d5d76-7d10-4af1-8689-77dd76070856-kube-api-access-v6n9q" pod "0e8d5d76-7d10-4af1-8689-77dd76070856" (UID: "0e8d5d76-7d10-4af1-8689-77dd76070856"). InnerVolumeSpecName "kube-api-access-v6n9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:01.197645 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.197613    4924 generic.go:356] "Generic (PLEG): container finished" podID="0e8d5d76-7d10-4af1-8689-77dd76070856" containerID="2be0add783dbbcab1079ffc5bf984b5619a3ebb21434e38a630b44c55682e773" exitCode=0
Oct 30 19:16:01.197856 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.197666    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x"
Oct 30 19:16:01.197856 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.197698    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x" event={"ID":"0e8d5d76-7d10-4af1-8689-77dd76070856","Type":"ContainerDied","Data":"2be0add783dbbcab1079ffc5bf984b5619a3ebb21434e38a630b44c55682e773"}
Oct 30 19:16:01.197856 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.197768    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x" event={"ID":"0e8d5d76-7d10-4af1-8689-77dd76070856","Type":"ContainerDied","Data":"88c5c56a1f03ebd6e345274b290ddf0074a5bc934235da4ae630369310bd3b30"}
Oct 30 19:16:01.197856 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.197803    4924 scope.go:122] "RemoveContainer" containerID="2be0add783dbbcab1079ffc5bf984b5619a3ebb21434e38a630b44c55682e773"
Oct 30 19:16:01.199428 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.199204    4924 generic.go:356] "Generic (PLEG): container finished" podID="9326b21d-3f94-4d9c-99f5-93e96de020bf" containerID="aa6bdf60801d333c730aeedce620826bb0b842d2457a5830cab43b4aaa74970f" exitCode=0
Oct 30 19:16:01.199428 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.199225    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f" event={"ID":"9326b21d-3f94-4d9c-99f5-93e96de020bf","Type":"ContainerDied","Data":"aa6bdf60801d333c730aeedce620826bb0b842d2457a5830cab43b4aaa74970f"}
Oct 30 19:16:01.199428 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.199259    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f"
Oct 30 19:16:01.199428 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.199274    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f" event={"ID":"9326b21d-3f94-4d9c-99f5-93e96de020bf","Type":"ContainerDied","Data":"212e7497dd1264957b1cff3003174db0e264574e8ee4ccb3a2a1c1ac312f2baf"}
Oct 30 19:16:01.204730 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.204702    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6n9q\" (UniqueName: \"kubernetes.io/projected/0e8d5d76-7d10-4af1-8689-77dd76070856-kube-api-access-v6n9q\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.204824 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.204736    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8d5d76-7d10-4af1-8689-77dd76070856-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:01.209717 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.209697    4924 scope.go:122] "RemoveContainer" containerID="2be0add783dbbcab1079ffc5bf984b5619a3ebb21434e38a630b44c55682e773"
Oct 30 19:16:01.210002 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:16:01.209980    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2be0add783dbbcab1079ffc5bf984b5619a3ebb21434e38a630b44c55682e773\": container with ID starting with 2be0add783dbbcab1079ffc5bf984b5619a3ebb21434e38a630b44c55682e773 not found: ID does not exist" containerID="2be0add783dbbcab1079ffc5bf984b5619a3ebb21434e38a630b44c55682e773"
Oct 30 19:16:01.210087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.210010    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be0add783dbbcab1079ffc5bf984b5619a3ebb21434e38a630b44c55682e773"} err="failed to get container status \"2be0add783dbbcab1079ffc5bf984b5619a3ebb21434e38a630b44c55682e773\": rpc error: code = NotFound desc = could not find container \"2be0add783dbbcab1079ffc5bf984b5619a3ebb21434e38a630b44c55682e773\": container with ID starting with 2be0add783dbbcab1079ffc5bf984b5619a3ebb21434e38a630b44c55682e773 not found: ID does not exist"
Oct 30 19:16:01.210087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.210025    4924 scope.go:122] "RemoveContainer" containerID="aa6bdf60801d333c730aeedce620826bb0b842d2457a5830cab43b4aaa74970f"
Oct 30 19:16:01.219128 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.219109    4924 scope.go:122] "RemoveContainer" containerID="aa6bdf60801d333c730aeedce620826bb0b842d2457a5830cab43b4aaa74970f"
Oct 30 19:16:01.219377 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:16:01.219356    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa6bdf60801d333c730aeedce620826bb0b842d2457a5830cab43b4aaa74970f\": container with ID starting with aa6bdf60801d333c730aeedce620826bb0b842d2457a5830cab43b4aaa74970f not found: ID does not exist" containerID="aa6bdf60801d333c730aeedce620826bb0b842d2457a5830cab43b4aaa74970f"
Oct 30 19:16:01.219439 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.219387    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa6bdf60801d333c730aeedce620826bb0b842d2457a5830cab43b4aaa74970f"} err="failed to get container status \"aa6bdf60801d333c730aeedce620826bb0b842d2457a5830cab43b4aaa74970f\": rpc error: code = NotFound desc = could not find container \"aa6bdf60801d333c730aeedce620826bb0b842d2457a5830cab43b4aaa74970f\": container with ID starting with aa6bdf60801d333c730aeedce620826bb0b842d2457a5830cab43b4aaa74970f not found: ID does not exist"
Oct 30 19:16:01.359315 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.359230    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x"]
Oct 30 19:16:01.379992 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.379961    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f3a0e7-67bdc6fdff-5mn8x"]
Oct 30 19:16:01.495355 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.495320    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f"]
Oct 30 19:16:01.503515 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:01.503485    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f3a0e7-6955d95984-hz99f"]
Oct 30 19:16:02.222417 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:02.222374    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:02.836202 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:02.836168    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e8d5d76-7d10-4af1-8689-77dd76070856" path="/var/lib/kubelet/pods/0e8d5d76-7d10-4af1-8689-77dd76070856/volumes"
Oct 30 19:16:02.836650 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:02.836636    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9326b21d-3f94-4d9c-99f5-93e96de020bf" path="/var/lib/kubelet/pods/9326b21d-3f94-4d9c-99f5-93e96de020bf/volumes"
Oct 30 19:16:02.877251 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:02.877208    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:05.444750 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:05.444711    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs" podUID="156d3474-d19a-4022-9a3c-88b93e53b527" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:16:07.242854 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:07.242821    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:08.494091 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:08.494035    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:08.694802 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:08.694769    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:09.691758 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:09.691722    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:11.558399 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:11.558343    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:12.741758 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:12.741723    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:13.119161 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:13.119121    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:13.405100 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:13.405001    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:13.605087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:13.600855    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk"]
Oct 30 19:16:13.640845 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:13.640809    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-649d16-fhxkk"]
Oct 30 19:16:14.131884 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:14.131813    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:14.393212 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:14.393055    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:14.668428 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:14.668301    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:14.835322 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:14.835289    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32b17e16-c7c7-4643-be8d-063422f810b0" path="/var/lib/kubelet/pods/32b17e16-c7c7-4643-be8d-063422f810b0/volumes"
Oct 30 19:16:14.927854 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:14.927777    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq"]
Oct 30 19:16:14.937258 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:14.937218    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:14.974474 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:14.974445    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8e31a7-rcrfq"]
Oct 30 19:16:15.444641 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:15.444605    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs" podUID="156d3474-d19a-4022-9a3c-88b93e53b527" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:16:15.452458 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:15.452421    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:15.712086 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:16:15.711994    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:16:15.948963 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:15.948937    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs"
Oct 30 19:16:16.026198 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:16.026114    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/156d3474-d19a-4022-9a3c-88b93e53b527-kube-api-access-6nmr7\" (UniqueName: \"kubernetes.io/projected/156d3474-d19a-4022-9a3c-88b93e53b527-kube-api-access-6nmr7\") pod \"156d3474-d19a-4022-9a3c-88b93e53b527\" (UID: \"156d3474-d19a-4022-9a3c-88b93e53b527\") "
Oct 30 19:16:16.026198 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:16.026150    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/156d3474-d19a-4022-9a3c-88b93e53b527-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156d3474-d19a-4022-9a3c-88b93e53b527-plugins-config-volume\") pod \"156d3474-d19a-4022-9a3c-88b93e53b527\" (UID: \"156d3474-d19a-4022-9a3c-88b93e53b527\") "
Oct 30 19:16:16.026621 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:16.026590    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/156d3474-d19a-4022-9a3c-88b93e53b527-plugins-config-volume" pod "156d3474-d19a-4022-9a3c-88b93e53b527" (UID: "156d3474-d19a-4022-9a3c-88b93e53b527"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:16.028463 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:16.028434    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156d3474-d19a-4022-9a3c-88b93e53b527-kube-api-access-6nmr7" pod "156d3474-d19a-4022-9a3c-88b93e53b527" (UID: "156d3474-d19a-4022-9a3c-88b93e53b527"). InnerVolumeSpecName "kube-api-access-6nmr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:16.127550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:16.127505    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nmr7\" (UniqueName: \"kubernetes.io/projected/156d3474-d19a-4022-9a3c-88b93e53b527-kube-api-access-6nmr7\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:16.127550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:16.127541    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156d3474-d19a-4022-9a3c-88b93e53b527-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:16.317244 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:16.317149    4924 generic.go:356] "Generic (PLEG): container finished" podID="156d3474-d19a-4022-9a3c-88b93e53b527" containerID="a738ae0497f8e4bd70d0d7068ee0873ba6e2c610f42eb86962fd63ef69bbbbdc" exitCode=0
Oct 30 19:16:16.317244 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:16.317185    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs" event={"ID":"156d3474-d19a-4022-9a3c-88b93e53b527","Type":"ContainerDied","Data":"a738ae0497f8e4bd70d0d7068ee0873ba6e2c610f42eb86962fd63ef69bbbbdc"}
Oct 30 19:16:16.317244 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:16.317206    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs"
Oct 30 19:16:16.317516 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:16.317254    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs" event={"ID":"156d3474-d19a-4022-9a3c-88b93e53b527","Type":"ContainerDied","Data":"c07c8bcf98f42393999accf69069e5bd9a7fcd143e5c95324cb4869ffc6f5109"}
Oct 30 19:16:16.317516 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:16.317322    4924 scope.go:122] "RemoveContainer" containerID="a738ae0497f8e4bd70d0d7068ee0873ba6e2c610f42eb86962fd63ef69bbbbdc"
Oct 30 19:16:16.327291 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:16.327272    4924 scope.go:122] "RemoveContainer" containerID="a738ae0497f8e4bd70d0d7068ee0873ba6e2c610f42eb86962fd63ef69bbbbdc"
Oct 30 19:16:16.327558 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:16:16.327535    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a738ae0497f8e4bd70d0d7068ee0873ba6e2c610f42eb86962fd63ef69bbbbdc\": container with ID starting with a738ae0497f8e4bd70d0d7068ee0873ba6e2c610f42eb86962fd63ef69bbbbdc not found: ID does not exist" containerID="a738ae0497f8e4bd70d0d7068ee0873ba6e2c610f42eb86962fd63ef69bbbbdc"
Oct 30 19:16:16.327630 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:16.327569    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a738ae0497f8e4bd70d0d7068ee0873ba6e2c610f42eb86962fd63ef69bbbbdc"} err="failed to get container status \"a738ae0497f8e4bd70d0d7068ee0873ba6e2c610f42eb86962fd63ef69bbbbdc\": rpc error: code = NotFound desc = could not find container \"a738ae0497f8e4bd70d0d7068ee0873ba6e2c610f42eb86962fd63ef69bbbbdc\": container with ID starting with a738ae0497f8e4bd70d0d7068ee0873ba6e2c610f42eb86962fd63ef69bbbbdc not found: ID does not exist"
Oct 30 19:16:16.475633 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:16.475599    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs"]
Oct 30 19:16:16.522128 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:16.522097    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a1617e-89d8c96c4-t5vbs"]
Oct 30 19:16:16.834931 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:16.834898    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="156d3474-d19a-4022-9a3c-88b93e53b527" path="/var/lib/kubelet/pods/156d3474-d19a-4022-9a3c-88b93e53b527/volumes"
Oct 30 19:16:16.835412 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:16.835396    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58181eb5-380f-49a1-bd72-b543779e9410" path="/var/lib/kubelet/pods/58181eb5-380f-49a1-bd72-b543779e9410/volumes"
Oct 30 19:16:19.980242 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:19.980014    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:20.171393 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:20.171363    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:20.373425 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:20.373389    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:21.684295 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:21.684259    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:22.266211 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:22.265694    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:22.524222 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:22.524054    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:22.842833 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:22.842746    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:23.440679 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:23.440647    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:23.746645 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:23.746559    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:23.975828 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:23.975790    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:24.080238 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:24.080162    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"]
Oct 30 19:16:24.080406 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:24.080385    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk" podUID="34498bbd-c6eb-48b9-8003-d863dd955292" containerName="istio-proxy" containerID="cri-o://4b1ce5947ac907be160170c20aec19d95e72fab48fb600c92f2199961eaafd3f" gracePeriod=30
Oct 30 19:16:24.221858 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:24.221826    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:24.533253 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:24.533221    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:24.825652 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:24.825557    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:25.312660 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.312619    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:25.334735 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.334701    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9"]
Oct 30 19:16:25.335274 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.335259    4924 cpu_manager.go:414] "containerMap: removing container" podUID="9918ebe6-aa19-488e-b259-e6201f326650" containerName="payload-pre-processing"
Oct 30 19:16:25.335335 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.335275    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9918ebe6-aa19-488e-b259-e6201f326650" containerName="payload-pre-processing"
Oct 30 19:16:25.335335 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.335295    4924 cpu_manager.go:414] "containerMap: removing container" podUID="bdc8df4b-8c94-40b8-b4be-80d1432d87de" containerName="main"
Oct 30 19:16:25.335335 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.335300    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdc8df4b-8c94-40b8-b4be-80d1432d87de" containerName="main"
Oct 30 19:16:25.335335 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.335310    4924 cpu_manager.go:414] "containerMap: removing container" podUID="156d3474-d19a-4022-9a3c-88b93e53b527" containerName="payload-processing"
Oct 30 19:16:25.335335 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.335315    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="156d3474-d19a-4022-9a3c-88b93e53b527" containerName="payload-processing"
Oct 30 19:16:25.335335 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.335323    4924 cpu_manager.go:414] "containerMap: removing container" podUID="9326b21d-3f94-4d9c-99f5-93e96de020bf" containerName="payload-processing"
Oct 30 19:16:25.335335 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.335327    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9326b21d-3f94-4d9c-99f5-93e96de020bf" containerName="payload-processing"
Oct 30 19:16:25.335561 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.335338    4924 cpu_manager.go:414] "containerMap: removing container" podUID="955e98b9-864b-4636-9511-8b69f981eaa7" containerName="main"
Oct 30 19:16:25.335561 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.335344    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="955e98b9-864b-4636-9511-8b69f981eaa7" containerName="main"
Oct 30 19:16:25.335561 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.335362    4924 cpu_manager.go:414] "containerMap: removing container" podUID="0e8d5d76-7d10-4af1-8689-77dd76070856" containerName="payload-pre-processing"
Oct 30 19:16:25.335561 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.335368    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e8d5d76-7d10-4af1-8689-77dd76070856" containerName="payload-pre-processing"
Oct 30 19:16:25.335561 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.335379    4924 cpu_manager.go:414] "containerMap: removing container" podUID="e9720622-6440-4232-8855-ae46ec5bf434" containerName="payload-processing"
Oct 30 19:16:25.335561 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.335383    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9720622-6440-4232-8855-ae46ec5bf434" containerName="payload-processing"
Oct 30 19:16:25.335561 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.335401    4924 cpu_manager.go:414] "containerMap: removing container" podUID="e1a3b34a-c6c5-483a-a419-784a3e5c95a5" containerName="revoke-keys"
Oct 30 19:16:25.335561 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.335405    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a3b34a-c6c5-483a-a419-784a3e5c95a5" containerName="revoke-keys"
Oct 30 19:16:25.335561 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.335487    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="9918ebe6-aa19-488e-b259-e6201f326650" containerName="payload-pre-processing"
Oct 30 19:16:25.335561 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.335496    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdc8df4b-8c94-40b8-b4be-80d1432d87de" containerName="main"
Oct 30 19:16:25.335561 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.335503    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="156d3474-d19a-4022-9a3c-88b93e53b527" containerName="payload-processing"
Oct 30 19:16:25.335561 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.335508    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="9326b21d-3f94-4d9c-99f5-93e96de020bf" containerName="payload-processing"
Oct 30 19:16:25.335561 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.335515    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="955e98b9-864b-4636-9511-8b69f981eaa7" containerName="main"
Oct 30 19:16:25.335561 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.335523    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e8d5d76-7d10-4af1-8689-77dd76070856" containerName="payload-pre-processing"
Oct 30 19:16:25.335561 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.335531    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9720622-6440-4232-8855-ae46ec5bf434" containerName="payload-processing"
Oct 30 19:16:25.335561 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.335537    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a3b34a-c6c5-483a-a419-784a3e5c95a5" containerName="revoke-keys"
Oct 30 19:16:25.340763 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.340743    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9"
Oct 30 19:16:25.395108 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.394338    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9"]
Oct 30 19:16:25.436327 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.436299    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:16:25.436498 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.436365    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:16:25.436498 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.436467    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:16:25.437324 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.437306    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:16:25.451274 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.451245    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a245bb-3de4-43a8-a017-e390e1092309-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9\" (UID: \"38a245bb-3de4-43a8-a017-e390e1092309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9"
Oct 30 19:16:25.451369 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.451329    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4h78\" (UniqueName: \"kubernetes.io/projected/38a245bb-3de4-43a8-a017-e390e1092309-kube-api-access-r4h78\") pod \"maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9\" (UID: \"38a245bb-3de4-43a8-a017-e390e1092309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9"
Oct 30 19:16:25.451614 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.451593    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a245bb-3de4-43a8-a017-e390e1092309-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9\" (UID: \"38a245bb-3de4-43a8-a017-e390e1092309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9"
Oct 30 19:16:25.553865 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.553820    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a245bb-3de4-43a8-a017-e390e1092309-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9\" (UID: \"38a245bb-3de4-43a8-a017-e390e1092309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9"
Oct 30 19:16:25.554128 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.554005    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a245bb-3de4-43a8-a017-e390e1092309-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9\" (UID: \"38a245bb-3de4-43a8-a017-e390e1092309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9"
Oct 30 19:16:25.554218 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.554126    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4h78\" (UniqueName: \"kubernetes.io/projected/38a245bb-3de4-43a8-a017-e390e1092309-kube-api-access-r4h78\") pod \"maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9\" (UID: \"38a245bb-3de4-43a8-a017-e390e1092309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9"
Oct 30 19:16:25.555192 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.555161    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a245bb-3de4-43a8-a017-e390e1092309-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9\" (UID: \"38a245bb-3de4-43a8-a017-e390e1092309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9"
Oct 30 19:16:25.559289 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.559263    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a245bb-3de4-43a8-a017-e390e1092309-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9\" (UID: \"38a245bb-3de4-43a8-a017-e390e1092309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9"
Oct 30 19:16:25.628377 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.628348    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4h78\" (UniqueName: \"kubernetes.io/projected/38a245bb-3de4-43a8-a017-e390e1092309-kube-api-access-r4h78\") pod \"maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9\" (UID: \"38a245bb-3de4-43a8-a017-e390e1092309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9"
Oct 30 19:16:25.657487 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.657460    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9"
Oct 30 19:16:25.885482 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:25.885383    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:26.046728 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:26.046696    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9"]
Oct 30 19:16:26.047440 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:16:26.047399    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a245bb_3de4_43a8_a017_e390e1092309.slice/crio-c1e0f9961f6f76098f1f0497c8aa07e2482f8fee2081fe8fd35ca38e09fcd303 WatchSource:0}: Error finding container c1e0f9961f6f76098f1f0497c8aa07e2482f8fee2081fe8fd35ca38e09fcd303: Status 404 returned error can't find the container with id c1e0f9961f6f76098f1f0497c8aa07e2482f8fee2081fe8fd35ca38e09fcd303
Oct 30 19:16:26.272404 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:26.272363    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:26.414676 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:26.414642    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9" event={"ID":"38a245bb-3de4-43a8-a017-e390e1092309","Type":"ContainerStarted","Data":"1faaa1146693c3f88a3cb56132992e250d08510ae7b30c5b28459ba4fa3225bf"}
Oct 30 19:16:26.415181 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:26.414696    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9" event={"ID":"38a245bb-3de4-43a8-a017-e390e1092309","Type":"ContainerStarted","Data":"c1e0f9961f6f76098f1f0497c8aa07e2482f8fee2081fe8fd35ca38e09fcd303"}
Oct 30 19:16:26.470228 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:26.470126    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:26.524818 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:26.524765    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9" podStartSLOduration=1.5247508010000002 podStartE2EDuration="1.524750801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:26.522495703 +0000 UTC m=+2387.799546160" watchObservedRunningTime="2026-10-30 19:16:26.524750801 +0000 UTC m=+2387.801801245"
Oct 30 19:16:26.654741 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:26.654706    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:26.731999 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:26.731918    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87"]
Oct 30 19:16:26.735247 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:26.735224    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f80ac-9rv87"]
Oct 30 19:16:26.896307 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:26.896268    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51b9a4f5-d2e7-462e-ab62-03ef1a125d89" path="/var/lib/kubelet/pods/51b9a4f5-d2e7-462e-ab62-03ef1a125d89/volumes"
Oct 30 19:16:27.150821 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.150785    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb621801-s69jf"]
Oct 30 19:16:27.155024 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.155003    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb621801-s69jf"
Oct 30 19:16:27.183582 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.183560    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/854e149f-ec52-42cd-8a49-ac4943ee6163-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb621801-s69jf\" (UID: \"854e149f-ec52-42cd-8a49-ac4943ee6163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb621801-s69jf"
Oct 30 19:16:27.183701 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.183614    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncl66\" (UniqueName: \"kubernetes.io/projected/854e149f-ec52-42cd-8a49-ac4943ee6163-kube-api-access-ncl66\") pod \"maas-api-revoke-sub-cb621801-s69jf\" (UID: \"854e149f-ec52-42cd-8a49-ac4943ee6163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb621801-s69jf"
Oct 30 19:16:27.183870 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.183850    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/854e149f-ec52-42cd-8a49-ac4943ee6163-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb621801-s69jf\" (UID: \"854e149f-ec52-42cd-8a49-ac4943ee6163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb621801-s69jf"
Oct 30 19:16:27.209702 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.209675    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb621801-s69jf"]
Oct 30 19:16:27.245680 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.245617    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a469b055-zdg5v"]
Oct 30 19:16:27.249800 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.249779    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a469b055-zdg5v"
Oct 30 19:16:27.255226 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.255205    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:27.285420 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.285388    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/854e149f-ec52-42cd-8a49-ac4943ee6163-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb621801-s69jf\" (UID: \"854e149f-ec52-42cd-8a49-ac4943ee6163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb621801-s69jf"
Oct 30 19:16:27.285519 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.285451    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f730bf-6bd9-45f3-b45c-e35ebff3cb39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a469b055-zdg5v\" (UID: \"37f730bf-6bd9-45f3-b45c-e35ebff3cb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a469b055-zdg5v"
Oct 30 19:16:27.285519 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.285480    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/854e149f-ec52-42cd-8a49-ac4943ee6163-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb621801-s69jf\" (UID: \"854e149f-ec52-42cd-8a49-ac4943ee6163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb621801-s69jf"
Oct 30 19:16:27.285519 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.285514    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qctgt\" (UniqueName: \"kubernetes.io/projected/37f730bf-6bd9-45f3-b45c-e35ebff3cb39-kube-api-access-qctgt\") pod \"maas-api-revoke-sub-a469b055-zdg5v\" (UID: \"37f730bf-6bd9-45f3-b45c-e35ebff3cb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a469b055-zdg5v"
Oct 30 19:16:27.285662 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.285639    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncl66\" (UniqueName: \"kubernetes.io/projected/854e149f-ec52-42cd-8a49-ac4943ee6163-kube-api-access-ncl66\") pod \"maas-api-revoke-sub-cb621801-s69jf\" (UID: \"854e149f-ec52-42cd-8a49-ac4943ee6163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb621801-s69jf"
Oct 30 19:16:27.285710 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.285695    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f730bf-6bd9-45f3-b45c-e35ebff3cb39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a469b055-zdg5v\" (UID: \"37f730bf-6bd9-45f3-b45c-e35ebff3cb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a469b055-zdg5v"
Oct 30 19:16:27.286059 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.286042    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/854e149f-ec52-42cd-8a49-ac4943ee6163-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb621801-s69jf\" (UID: \"854e149f-ec52-42cd-8a49-ac4943ee6163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb621801-s69jf"
Oct 30 19:16:27.287910 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.287887    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/854e149f-ec52-42cd-8a49-ac4943ee6163-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb621801-s69jf\" (UID: \"854e149f-ec52-42cd-8a49-ac4943ee6163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb621801-s69jf"
Oct 30 19:16:27.307468 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.307446    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a469b055-zdg5v"]
Oct 30 19:16:27.341383 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.341346    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncl66\" (UniqueName: \"kubernetes.io/projected/854e149f-ec52-42cd-8a49-ac4943ee6163-kube-api-access-ncl66\") pod \"maas-api-revoke-sub-cb621801-s69jf\" (UID: \"854e149f-ec52-42cd-8a49-ac4943ee6163\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb621801-s69jf"
Oct 30 19:16:27.386924 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.386891    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f730bf-6bd9-45f3-b45c-e35ebff3cb39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a469b055-zdg5v\" (UID: \"37f730bf-6bd9-45f3-b45c-e35ebff3cb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a469b055-zdg5v"
Oct 30 19:16:27.387123 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.386954    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qctgt\" (UniqueName: \"kubernetes.io/projected/37f730bf-6bd9-45f3-b45c-e35ebff3cb39-kube-api-access-qctgt\") pod \"maas-api-revoke-sub-a469b055-zdg5v\" (UID: \"37f730bf-6bd9-45f3-b45c-e35ebff3cb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a469b055-zdg5v"
Oct 30 19:16:27.387123 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.386991    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f730bf-6bd9-45f3-b45c-e35ebff3cb39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a469b055-zdg5v\" (UID: \"37f730bf-6bd9-45f3-b45c-e35ebff3cb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a469b055-zdg5v"
Oct 30 19:16:27.387749 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.387719    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f730bf-6bd9-45f3-b45c-e35ebff3cb39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a469b055-zdg5v\" (UID: \"37f730bf-6bd9-45f3-b45c-e35ebff3cb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a469b055-zdg5v"
Oct 30 19:16:27.389343 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.389320    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f730bf-6bd9-45f3-b45c-e35ebff3cb39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a469b055-zdg5v\" (UID: \"37f730bf-6bd9-45f3-b45c-e35ebff3cb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a469b055-zdg5v"
Oct 30 19:16:27.423586 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.423552    4924 generic.go:356] "Generic (PLEG): container finished" podID="38a245bb-3de4-43a8-a017-e390e1092309" containerID="1faaa1146693c3f88a3cb56132992e250d08510ae7b30c5b28459ba4fa3225bf" exitCode=0
Oct 30 19:16:27.424052 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.423642    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9" event={"ID":"38a245bb-3de4-43a8-a017-e390e1092309","Type":"ContainerDied","Data":"1faaa1146693c3f88a3cb56132992e250d08510ae7b30c5b28459ba4fa3225bf"}
Oct 30 19:16:27.446912 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.446880    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qctgt\" (UniqueName: \"kubernetes.io/projected/37f730bf-6bd9-45f3-b45c-e35ebff3cb39-kube-api-access-qctgt\") pod \"maas-api-revoke-sub-a469b055-zdg5v\" (UID: \"37f730bf-6bd9-45f3-b45c-e35ebff3cb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a469b055-zdg5v"
Oct 30 19:16:27.468322 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.468295    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb621801-s69jf"
Oct 30 19:16:27.563469 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.563270    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a469b055-zdg5v"
Oct 30 19:16:27.787595 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.787502    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:27.844795 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:16:27.844767    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod854e149f_ec52_42cd_8a49_ac4943ee6163.slice/crio-cfb7aaeeb4497cd75ffeeb927f16f15bdba9e6dcdf5b396a32c811db65662b0b WatchSource:0}: Error finding container cfb7aaeeb4497cd75ffeeb927f16f15bdba9e6dcdf5b396a32c811db65662b0b: Status 404 returned error can't find the container with id cfb7aaeeb4497cd75ffeeb927f16f15bdba9e6dcdf5b396a32c811db65662b0b
Oct 30 19:16:27.854182 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.854158    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb621801-s69jf"]
Oct 30 19:16:27.982979 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:16:27.982954    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f730bf_6bd9_45f3_b45c_e35ebff3cb39.slice/crio-ac1318df730dd1f775b229d9bc2fa97dd59b7c4e4fddfbc266c4add83a5961fc WatchSource:0}: Error finding container ac1318df730dd1f775b229d9bc2fa97dd59b7c4e4fddfbc266c4add83a5961fc: Status 404 returned error can't find the container with id ac1318df730dd1f775b229d9bc2fa97dd59b7c4e4fddfbc266c4add83a5961fc
Oct 30 19:16:27.991890 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:27.991855    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a469b055-zdg5v"]
Oct 30 19:16:28.138141 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:28.138108    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:28.256403 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:28.256370    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh"]
Oct 30 19:16:28.259493 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:28.259469    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3a0e7-qq8lh"]
Oct 30 19:16:28.431543 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:28.431442    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:28.433048 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:28.433020    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a469b055-zdg5v" event={"ID":"37f730bf-6bd9-45f3-b45c-e35ebff3cb39","Type":"ContainerStarted","Data":"dad5539a375292c5562b07a3011e9c51f81da5e4cc617af1a380f38db1d8424e"}
Oct 30 19:16:28.433217 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:28.433112    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a469b055-zdg5v" event={"ID":"37f730bf-6bd9-45f3-b45c-e35ebff3cb39","Type":"ContainerStarted","Data":"ac1318df730dd1f775b229d9bc2fa97dd59b7c4e4fddfbc266c4add83a5961fc"}
Oct 30 19:16:28.434825 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:28.434797    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb621801-s69jf" event={"ID":"854e149f-ec52-42cd-8a49-ac4943ee6163","Type":"ContainerStarted","Data":"6dbdc5fa3ca4a21a608e4f6ef5f869506abc70545ceba4b71c32897d1e86a661"}
Oct 30 19:16:28.434950 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:28.434849    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb621801-s69jf" event={"ID":"854e149f-ec52-42cd-8a49-ac4943ee6163","Type":"ContainerStarted","Data":"cfb7aaeeb4497cd75ffeeb927f16f15bdba9e6dcdf5b396a32c811db65662b0b"}
Oct 30 19:16:28.547642 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:28.547589    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a469b055-zdg5v" podStartSLOduration=1.547575509 podStartE2EDuration="1.547575509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:28.543609273 +0000 UTC m=+2389.820659719" watchObservedRunningTime="2026-10-30 19:16:28.547575509 +0000 UTC m=+2389.824625950"
Oct 30 19:16:28.583925 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:28.583896    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9"
Oct 30 19:16:28.658517 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:28.658466    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb621801-s69jf" podStartSLOduration=1.658450823 podStartE2EDuration="1.658450823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:28.655226174 +0000 UTC m=+2389.932276619" watchObservedRunningTime="2026-10-30 19:16:28.658450823 +0000 UTC m=+2389.935501267"
Oct 30 19:16:28.708203 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:28.708173    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38a245bb-3de4-43a8-a017-e390e1092309-kube-api-access-r4h78\" (UniqueName: \"kubernetes.io/projected/38a245bb-3de4-43a8-a017-e390e1092309-kube-api-access-r4h78\") pod \"38a245bb-3de4-43a8-a017-e390e1092309\" (UID: \"38a245bb-3de4-43a8-a017-e390e1092309\") "
Oct 30 19:16:28.708372 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:28.708221    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38a245bb-3de4-43a8-a017-e390e1092309-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a245bb-3de4-43a8-a017-e390e1092309-maas-api-xks-ca\") pod \"38a245bb-3de4-43a8-a017-e390e1092309\" (UID: \"38a245bb-3de4-43a8-a017-e390e1092309\") "
Oct 30 19:16:28.708372 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:28.708355    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38a245bb-3de4-43a8-a017-e390e1092309-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a245bb-3de4-43a8-a017-e390e1092309-maas-api-openshift-ca\") pod \"38a245bb-3de4-43a8-a017-e390e1092309\" (UID: \"38a245bb-3de4-43a8-a017-e390e1092309\") "
Oct 30 19:16:28.708791 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:28.708759    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38a245bb-3de4-43a8-a017-e390e1092309-maas-api-openshift-ca" pod "38a245bb-3de4-43a8-a017-e390e1092309" (UID: "38a245bb-3de4-43a8-a017-e390e1092309"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:28.709003 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:28.708982    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a245bb-3de4-43a8-a017-e390e1092309-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:28.710641 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:28.710488    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38a245bb-3de4-43a8-a017-e390e1092309-maas-api-xks-ca" pod "38a245bb-3de4-43a8-a017-e390e1092309" (UID: "38a245bb-3de4-43a8-a017-e390e1092309"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:28.711058 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:28.711030    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a245bb-3de4-43a8-a017-e390e1092309-kube-api-access-r4h78" pod "38a245bb-3de4-43a8-a017-e390e1092309" (UID: "38a245bb-3de4-43a8-a017-e390e1092309"). InnerVolumeSpecName "kube-api-access-r4h78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:28.797569 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:28.797509    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:28.810923 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:28.810887    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4h78\" (UniqueName: \"kubernetes.io/projected/38a245bb-3de4-43a8-a017-e390e1092309-kube-api-access-r4h78\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:28.810923 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:28.810926    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a245bb-3de4-43a8-a017-e390e1092309-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:28.835686 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:28.835650    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1a3b34a-c6c5-483a-a419-784a3e5c95a5" path="/var/lib/kubelet/pods/e1a3b34a-c6c5-483a-a419-784a3e5c95a5/volumes"
Oct 30 19:16:29.334776 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.334751    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:16:29.417983 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.417955    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-istio-envoy\") pod \"34498bbd-c6eb-48b9-8003-d863dd955292\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") "
Oct 30 19:16:29.418195 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.418037    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34498bbd-c6eb-48b9-8003-d863dd955292-kube-api-access-jq7r7\" (UniqueName: \"kubernetes.io/projected/34498bbd-c6eb-48b9-8003-d863dd955292-kube-api-access-jq7r7\") pod \"34498bbd-c6eb-48b9-8003-d863dd955292\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") "
Oct 30 19:16:29.418195 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.418125    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-credential-socket\") pod \"34498bbd-c6eb-48b9-8003-d863dd955292\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") "
Oct 30 19:16:29.418195 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.418191    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34498bbd-c6eb-48b9-8003-d863dd955292-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34498bbd-c6eb-48b9-8003-d863dd955292-istiod-ca-cert\") pod \"34498bbd-c6eb-48b9-8003-d863dd955292\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") "
Oct 30 19:16:29.418376 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.418248    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34498bbd-c6eb-48b9-8003-d863dd955292-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34498bbd-c6eb-48b9-8003-d863dd955292-istio-podinfo\") pod \"34498bbd-c6eb-48b9-8003-d863dd955292\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") "
Oct 30 19:16:29.418376 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.418332    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-istio-data\") pod \"34498bbd-c6eb-48b9-8003-d863dd955292\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") "
Oct 30 19:16:29.418483 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.418410    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-workload-socket\") pod \"34498bbd-c6eb-48b9-8003-d863dd955292\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") "
Oct 30 19:16:29.418483 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.418410    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-credential-socket" pod "34498bbd-c6eb-48b9-8003-d863dd955292" (UID: "34498bbd-c6eb-48b9-8003-d863dd955292"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:29.418483 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.418449    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-workload-certs\") pod \"34498bbd-c6eb-48b9-8003-d863dd955292\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") "
Oct 30 19:16:29.418641 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.418488    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34498bbd-c6eb-48b9-8003-d863dd955292-istio-token\" (UniqueName: \"kubernetes.io/projected/34498bbd-c6eb-48b9-8003-d863dd955292-istio-token\") pod \"34498bbd-c6eb-48b9-8003-d863dd955292\" (UID: \"34498bbd-c6eb-48b9-8003-d863dd955292\") "
Oct 30 19:16:29.418719 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.418598    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34498bbd-c6eb-48b9-8003-d863dd955292-istiod-ca-cert" pod "34498bbd-c6eb-48b9-8003-d863dd955292" (UID: "34498bbd-c6eb-48b9-8003-d863dd955292"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:29.418719 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.418676    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-workload-socket" pod "34498bbd-c6eb-48b9-8003-d863dd955292" (UID: "34498bbd-c6eb-48b9-8003-d863dd955292"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:29.418843 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.418769    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-istio-data" pod "34498bbd-c6eb-48b9-8003-d863dd955292" (UID: "34498bbd-c6eb-48b9-8003-d863dd955292"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:29.418944 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.418920    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-workload-certs" pod "34498bbd-c6eb-48b9-8003-d863dd955292" (UID: "34498bbd-c6eb-48b9-8003-d863dd955292"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:29.419173 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.419150    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:29.419280 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.419180    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:29.419280 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.419197    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:29.419280 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.419213    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34498bbd-c6eb-48b9-8003-d863dd955292-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:29.420352 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.420330    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34498bbd-c6eb-48b9-8003-d863dd955292-kube-api-access-jq7r7" pod "34498bbd-c6eb-48b9-8003-d863dd955292" (UID: "34498bbd-c6eb-48b9-8003-d863dd955292"). InnerVolumeSpecName "kube-api-access-jq7r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:29.420471 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.420349    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-istio-envoy" pod "34498bbd-c6eb-48b9-8003-d863dd955292" (UID: "34498bbd-c6eb-48b9-8003-d863dd955292"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:29.420471 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.420419    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34498bbd-c6eb-48b9-8003-d863dd955292-istio-podinfo" pod "34498bbd-c6eb-48b9-8003-d863dd955292" (UID: "34498bbd-c6eb-48b9-8003-d863dd955292"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:16:29.420695 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.420681    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34498bbd-c6eb-48b9-8003-d863dd955292-istio-token" pod "34498bbd-c6eb-48b9-8003-d863dd955292" (UID: "34498bbd-c6eb-48b9-8003-d863dd955292"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:29.445446 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.445419    4924 generic.go:356] "Generic (PLEG): container finished" podID="34498bbd-c6eb-48b9-8003-d863dd955292" containerID="4b1ce5947ac907be160170c20aec19d95e72fab48fb600c92f2199961eaafd3f" exitCode=0
Oct 30 19:16:29.445821 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.445472    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"
Oct 30 19:16:29.445821 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.445467    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk" event={"ID":"34498bbd-c6eb-48b9-8003-d863dd955292","Type":"ContainerDied","Data":"4b1ce5947ac907be160170c20aec19d95e72fab48fb600c92f2199961eaafd3f"}
Oct 30 19:16:29.445821 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.445534    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk" event={"ID":"34498bbd-c6eb-48b9-8003-d863dd955292","Type":"ContainerDied","Data":"fa07b98ddde96d6148650d455077c001891dbdb32c03a506eaea195ddaedc763"}
Oct 30 19:16:29.445821 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.445569    4924 scope.go:122] "RemoveContainer" containerID="4b1ce5947ac907be160170c20aec19d95e72fab48fb600c92f2199961eaafd3f"
Oct 30 19:16:29.447656 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.447629    4924 generic.go:356] "Generic (PLEG): container finished" podID="37f730bf-6bd9-45f3-b45c-e35ebff3cb39" containerID="dad5539a375292c5562b07a3011e9c51f81da5e4cc617af1a380f38db1d8424e" exitCode=0
Oct 30 19:16:29.447773 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.447668    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a469b055-zdg5v" event={"ID":"37f730bf-6bd9-45f3-b45c-e35ebff3cb39","Type":"ContainerDied","Data":"dad5539a375292c5562b07a3011e9c51f81da5e4cc617af1a380f38db1d8424e"}
Oct 30 19:16:29.449429 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.449400    4924 generic.go:356] "Generic (PLEG): container finished" podID="854e149f-ec52-42cd-8a49-ac4943ee6163" containerID="6dbdc5fa3ca4a21a608e4f6ef5f869506abc70545ceba4b71c32897d1e86a661" exitCode=0
Oct 30 19:16:29.449516 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.449476    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb621801-s69jf" event={"ID":"854e149f-ec52-42cd-8a49-ac4943ee6163","Type":"ContainerDied","Data":"6dbdc5fa3ca4a21a608e4f6ef5f869506abc70545ceba4b71c32897d1e86a661"}
Oct 30 19:16:29.451220 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.451135    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9"
Oct 30 19:16:29.451220 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.451143    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9" event={"ID":"38a245bb-3de4-43a8-a017-e390e1092309","Type":"ContainerDied","Data":"c1e0f9961f6f76098f1f0497c8aa07e2482f8fee2081fe8fd35ca38e09fcd303"}
Oct 30 19:16:29.451220 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.451191    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1e0f9961f6f76098f1f0497c8aa07e2482f8fee2081fe8fd35ca38e09fcd303"
Oct 30 19:16:29.458701 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.458685    4924 scope.go:122] "RemoveContainer" containerID="90b30198a6b8abc78e069f1c58737044cc9e9edc6e57e2517fd1e29f4a6a4921"
Oct 30 19:16:29.462856 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.462835    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:29.468794 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.468776    4924 scope.go:122] "RemoveContainer" containerID="4b1ce5947ac907be160170c20aec19d95e72fab48fb600c92f2199961eaafd3f"
Oct 30 19:16:29.469043 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:16:29.469023    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b1ce5947ac907be160170c20aec19d95e72fab48fb600c92f2199961eaafd3f\": container with ID starting with 4b1ce5947ac907be160170c20aec19d95e72fab48fb600c92f2199961eaafd3f not found: ID does not exist" containerID="4b1ce5947ac907be160170c20aec19d95e72fab48fb600c92f2199961eaafd3f"
Oct 30 19:16:29.469097 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.469052    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b1ce5947ac907be160170c20aec19d95e72fab48fb600c92f2199961eaafd3f"} err="failed to get container status \"4b1ce5947ac907be160170c20aec19d95e72fab48fb600c92f2199961eaafd3f\": rpc error: code = NotFound desc = could not find container \"4b1ce5947ac907be160170c20aec19d95e72fab48fb600c92f2199961eaafd3f\": container with ID starting with 4b1ce5947ac907be160170c20aec19d95e72fab48fb600c92f2199961eaafd3f not found: ID does not exist"
Oct 30 19:16:29.469097 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.469083    4924 scope.go:122] "RemoveContainer" containerID="90b30198a6b8abc78e069f1c58737044cc9e9edc6e57e2517fd1e29f4a6a4921"
Oct 30 19:16:29.469326 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:16:29.469309    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90b30198a6b8abc78e069f1c58737044cc9e9edc6e57e2517fd1e29f4a6a4921\": container with ID starting with 90b30198a6b8abc78e069f1c58737044cc9e9edc6e57e2517fd1e29f4a6a4921 not found: ID does not exist" containerID="90b30198a6b8abc78e069f1c58737044cc9e9edc6e57e2517fd1e29f4a6a4921"
Oct 30 19:16:29.469361 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.469333    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b30198a6b8abc78e069f1c58737044cc9e9edc6e57e2517fd1e29f4a6a4921"} err="failed to get container status \"90b30198a6b8abc78e069f1c58737044cc9e9edc6e57e2517fd1e29f4a6a4921\": rpc error: code = NotFound desc = could not find container \"90b30198a6b8abc78e069f1c58737044cc9e9edc6e57e2517fd1e29f4a6a4921\": container with ID starting with 90b30198a6b8abc78e069f1c58737044cc9e9edc6e57e2517fd1e29f4a6a4921 not found: ID does not exist"
Oct 30 19:16:29.520550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.520460    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34498bbd-c6eb-48b9-8003-d863dd955292-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:29.520550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.520489    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:29.520550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.520500    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq7r7\" (UniqueName: \"kubernetes.io/projected/34498bbd-c6eb-48b9-8003-d863dd955292-kube-api-access-jq7r7\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:29.520550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.520509    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34498bbd-c6eb-48b9-8003-d863dd955292-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:29.520550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.520518    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34498bbd-c6eb-48b9-8003-d863dd955292-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:29.751766 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.751738    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:29.778294 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.778218    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"]
Oct 30 19:16:29.898666 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.897132    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"]
Oct 30 19:16:29.902141 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.902116    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-8e31a7-openshift-default-55878bddb9-q6zrk"]
Oct 30 19:16:29.959050 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:29.959022    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:30.638147 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:30.638117    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:30.649419 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:30.649390    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb621801-s69jf"
Oct 30 19:16:30.651378 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:30.651360    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a469b055-zdg5v"
Oct 30 19:16:30.735288 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:30.735216    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/854e149f-ec52-42cd-8a49-ac4943ee6163-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/854e149f-ec52-42cd-8a49-ac4943ee6163-maas-api-openshift-ca\") pod \"854e149f-ec52-42cd-8a49-ac4943ee6163\" (UID: \"854e149f-ec52-42cd-8a49-ac4943ee6163\") "
Oct 30 19:16:30.735446 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:30.735286    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/854e149f-ec52-42cd-8a49-ac4943ee6163-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/854e149f-ec52-42cd-8a49-ac4943ee6163-maas-api-xks-ca\") pod \"854e149f-ec52-42cd-8a49-ac4943ee6163\" (UID: \"854e149f-ec52-42cd-8a49-ac4943ee6163\") "
Oct 30 19:16:30.735446 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:30.735419    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37f730bf-6bd9-45f3-b45c-e35ebff3cb39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f730bf-6bd9-45f3-b45c-e35ebff3cb39-maas-api-xks-ca\") pod \"37f730bf-6bd9-45f3-b45c-e35ebff3cb39\" (UID: \"37f730bf-6bd9-45f3-b45c-e35ebff3cb39\") "
Oct 30 19:16:30.735588 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:30.735474    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/854e149f-ec52-42cd-8a49-ac4943ee6163-kube-api-access-ncl66\" (UniqueName: \"kubernetes.io/projected/854e149f-ec52-42cd-8a49-ac4943ee6163-kube-api-access-ncl66\") pod \"854e149f-ec52-42cd-8a49-ac4943ee6163\" (UID: \"854e149f-ec52-42cd-8a49-ac4943ee6163\") "
Oct 30 19:16:30.735647 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:30.735618    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37f730bf-6bd9-45f3-b45c-e35ebff3cb39-kube-api-access-qctgt\" (UniqueName: \"kubernetes.io/projected/37f730bf-6bd9-45f3-b45c-e35ebff3cb39-kube-api-access-qctgt\") pod \"37f730bf-6bd9-45f3-b45c-e35ebff3cb39\" (UID: \"37f730bf-6bd9-45f3-b45c-e35ebff3cb39\") "
Oct 30 19:16:30.735647 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:30.735625    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/854e149f-ec52-42cd-8a49-ac4943ee6163-maas-api-openshift-ca" pod "854e149f-ec52-42cd-8a49-ac4943ee6163" (UID: "854e149f-ec52-42cd-8a49-ac4943ee6163"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:30.735752 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:30.735675    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37f730bf-6bd9-45f3-b45c-e35ebff3cb39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f730bf-6bd9-45f3-b45c-e35ebff3cb39-maas-api-openshift-ca\") pod \"37f730bf-6bd9-45f3-b45c-e35ebff3cb39\" (UID: \"37f730bf-6bd9-45f3-b45c-e35ebff3cb39\") "
Oct 30 19:16:30.736149 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:30.736120    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f730bf-6bd9-45f3-b45c-e35ebff3cb39-maas-api-openshift-ca" pod "37f730bf-6bd9-45f3-b45c-e35ebff3cb39" (UID: "37f730bf-6bd9-45f3-b45c-e35ebff3cb39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:30.736271 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:30.736255    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37f730bf-6bd9-45f3-b45c-e35ebff3cb39-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:30.736318 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:30.736275    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/854e149f-ec52-42cd-8a49-ac4943ee6163-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:30.737624 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:30.737598    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37f730bf-6bd9-45f3-b45c-e35ebff3cb39-maas-api-xks-ca" pod "37f730bf-6bd9-45f3-b45c-e35ebff3cb39" (UID: "37f730bf-6bd9-45f3-b45c-e35ebff3cb39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:30.737760 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:30.737672    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f730bf-6bd9-45f3-b45c-e35ebff3cb39-kube-api-access-qctgt" pod "37f730bf-6bd9-45f3-b45c-e35ebff3cb39" (UID: "37f730bf-6bd9-45f3-b45c-e35ebff3cb39"). InnerVolumeSpecName "kube-api-access-qctgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:30.737944 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:30.737921    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/854e149f-ec52-42cd-8a49-ac4943ee6163-maas-api-xks-ca" pod "854e149f-ec52-42cd-8a49-ac4943ee6163" (UID: "854e149f-ec52-42cd-8a49-ac4943ee6163"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:30.738036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:30.738000    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/854e149f-ec52-42cd-8a49-ac4943ee6163-kube-api-access-ncl66" pod "854e149f-ec52-42cd-8a49-ac4943ee6163" (UID: "854e149f-ec52-42cd-8a49-ac4943ee6163"). InnerVolumeSpecName "kube-api-access-ncl66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:30.835946 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:30.835915    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34498bbd-c6eb-48b9-8003-d863dd955292" path="/var/lib/kubelet/pods/34498bbd-c6eb-48b9-8003-d863dd955292/volumes"
Oct 30 19:16:30.836934 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:30.836911    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/854e149f-ec52-42cd-8a49-ac4943ee6163-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:30.836934 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:30.836934    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37f730bf-6bd9-45f3-b45c-e35ebff3cb39-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:30.837124 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:30.836944    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncl66\" (UniqueName: \"kubernetes.io/projected/854e149f-ec52-42cd-8a49-ac4943ee6163-kube-api-access-ncl66\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:30.837124 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:30.836956    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qctgt\" (UniqueName: \"kubernetes.io/projected/37f730bf-6bd9-45f3-b45c-e35ebff3cb39-kube-api-access-qctgt\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:31.469119 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:31.469085    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a469b055-zdg5v"
Oct 30 19:16:31.469290 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:31.469087    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a469b055-zdg5v" event={"ID":"37f730bf-6bd9-45f3-b45c-e35ebff3cb39","Type":"ContainerDied","Data":"ac1318df730dd1f775b229d9bc2fa97dd59b7c4e4fddfbc266c4add83a5961fc"}
Oct 30 19:16:31.469290 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:31.469220    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac1318df730dd1f775b229d9bc2fa97dd59b7c4e4fddfbc266c4add83a5961fc"
Oct 30 19:16:31.470549 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:31.470526    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb621801-s69jf"
Oct 30 19:16:31.470549 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:31.470536    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb621801-s69jf" event={"ID":"854e149f-ec52-42cd-8a49-ac4943ee6163","Type":"ContainerDied","Data":"cfb7aaeeb4497cd75ffeeb927f16f15bdba9e6dcdf5b396a32c811db65662b0b"}
Oct 30 19:16:31.470708 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:31.470578    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfb7aaeeb4497cd75ffeeb927f16f15bdba9e6dcdf5b396a32c811db65662b0b"
Oct 30 19:16:33.308275 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:33.308237    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:33.698672 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:33.698617    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:33.912471 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:33.912441    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"]
Oct 30 19:16:33.912636 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:33.912617    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4" podUID="c81ff6d8-e46d-467e-b3fe-49d3e95b34f2" containerName="istio-proxy" containerID="cri-o://1912e82c51a53e10f18c7eb541e95ea34ed4712cdad45a9bee99d24ace3a00de" gracePeriod=30
Oct 30 19:16:33.949511 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:33.949441    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:34.203259 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:34.203180    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:36.730274 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:36.730241    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:36.825927 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:36.825894    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c"]
Oct 30 19:16:36.928456 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:36.928424    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4"]
Oct 30 19:16:36.938959 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:36.938918    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:37.039521 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:37.039444    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"]
Oct 30 19:16:37.039690 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:37.039670    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf" podUID="0c9fa098-37e6-48ad-ba03-1879f9017f7c" containerName="istio-proxy" containerID="cri-o://22b4c7af931de94782cbf1bde654e507b4be99e1a07af3ca9a39baa355e46197" gracePeriod=30
Oct 30 19:16:37.146628 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:37.146586    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:37.370309 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:37.370277    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:37.773612 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:37.773503    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:38.030264 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.030182    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p"]
Oct 30 19:16:38.031047 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.031022    4924 cpu_manager.go:414] "containerMap: removing container" podUID="34498bbd-c6eb-48b9-8003-d863dd955292" containerName="istio-proxy"
Oct 30 19:16:38.031104 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.031052    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34498bbd-c6eb-48b9-8003-d863dd955292" containerName="istio-proxy"
Oct 30 19:16:38.031104 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.031092    4924 cpu_manager.go:414] "containerMap: removing container" podUID="38a245bb-3de4-43a8-a017-e390e1092309" containerName="revoke-keys"
Oct 30 19:16:38.031104 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.031101    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38a245bb-3de4-43a8-a017-e390e1092309" containerName="revoke-keys"
Oct 30 19:16:38.031213 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.031126    4924 cpu_manager.go:414] "containerMap: removing container" podUID="854e149f-ec52-42cd-8a49-ac4943ee6163" containerName="revoke-keys"
Oct 30 19:16:38.031213 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.031133    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="854e149f-ec52-42cd-8a49-ac4943ee6163" containerName="revoke-keys"
Oct 30 19:16:38.031213 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.031155    4924 cpu_manager.go:414] "containerMap: removing container" podUID="37f730bf-6bd9-45f3-b45c-e35ebff3cb39" containerName="revoke-keys"
Oct 30 19:16:38.031213 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.031162    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f730bf-6bd9-45f3-b45c-e35ebff3cb39" containerName="revoke-keys"
Oct 30 19:16:38.031360 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.031346    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="34498bbd-c6eb-48b9-8003-d863dd955292" containerName="istio-proxy"
Oct 30 19:16:38.031400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.031365    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="38a245bb-3de4-43a8-a017-e390e1092309" containerName="revoke-keys"
Oct 30 19:16:38.031400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.031380    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="854e149f-ec52-42cd-8a49-ac4943ee6163" containerName="revoke-keys"
Oct 30 19:16:38.031400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.031390    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="37f730bf-6bd9-45f3-b45c-e35ebff3cb39" containerName="revoke-keys"
Oct 30 19:16:38.034622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.034602    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p"
Oct 30 19:16:38.089227 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.089196    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:16:38.093194 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.093155    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:16:38.097266 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.097241    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:16:38.097398 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.097311    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:16:38.098001 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.097980    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p"]
Oct 30 19:16:38.125858 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.125827    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7a23dbd-97df-45af-b79b-87ead5172a61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p\" (UID: \"f7a23dbd-97df-45af-b79b-87ead5172a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p"
Oct 30 19:16:38.126023 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.125958    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7a23dbd-97df-45af-b79b-87ead5172a61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p\" (UID: \"f7a23dbd-97df-45af-b79b-87ead5172a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p"
Oct 30 19:16:38.126178 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.126157    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hg6w\" (UniqueName: \"kubernetes.io/projected/f7a23dbd-97df-45af-b79b-87ead5172a61-kube-api-access-2hg6w\") pod \"maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p\" (UID: \"f7a23dbd-97df-45af-b79b-87ead5172a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p"
Oct 30 19:16:38.234087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.229944    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hg6w\" (UniqueName: \"kubernetes.io/projected/f7a23dbd-97df-45af-b79b-87ead5172a61-kube-api-access-2hg6w\") pod \"maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p\" (UID: \"f7a23dbd-97df-45af-b79b-87ead5172a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p"
Oct 30 19:16:38.234087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.230172    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7a23dbd-97df-45af-b79b-87ead5172a61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p\" (UID: \"f7a23dbd-97df-45af-b79b-87ead5172a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p"
Oct 30 19:16:38.234087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.230244    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7a23dbd-97df-45af-b79b-87ead5172a61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p\" (UID: \"f7a23dbd-97df-45af-b79b-87ead5172a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p"
Oct 30 19:16:38.234087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.231188    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7a23dbd-97df-45af-b79b-87ead5172a61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p\" (UID: \"f7a23dbd-97df-45af-b79b-87ead5172a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p"
Oct 30 19:16:38.260998 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.260927    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7a23dbd-97df-45af-b79b-87ead5172a61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p\" (UID: \"f7a23dbd-97df-45af-b79b-87ead5172a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p"
Oct 30 19:16:38.315020 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.314939    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hg6w\" (UniqueName: \"kubernetes.io/projected/f7a23dbd-97df-45af-b79b-87ead5172a61-kube-api-access-2hg6w\") pod \"maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p\" (UID: \"f7a23dbd-97df-45af-b79b-87ead5172a61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p"
Oct 30 19:16:38.348363 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.348331    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p"
Oct 30 19:16:38.762750 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:38.762722    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p"]
Oct 30 19:16:38.763873 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:16:38.763846    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a23dbd_97df_45af_b79b_87ead5172a61.slice/crio-7bc6ebf821e133b45c119cb08554100b2da5bc433e2cf2871cf8866a81c27be8 WatchSource:0}: Error finding container 7bc6ebf821e133b45c119cb08554100b2da5bc433e2cf2871cf8866a81c27be8: Status 404 returned error can't find the container with id 7bc6ebf821e133b45c119cb08554100b2da5bc433e2cf2871cf8866a81c27be8
Oct 30 19:16:39.060817 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.060777    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:39.219397 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.219370    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 19:16:39.277526 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.277126    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 19:16:39.280778 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.279808    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 19:16:39.370021 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.369984    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-data\") pod \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") "
Oct 30 19:16:39.370202 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.370079    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-token\" (UniqueName: \"kubernetes.io/projected/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-token\") pod \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") "
Oct 30 19:16:39.370202 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.370129    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-credential-socket\") pod \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") "
Oct 30 19:16:39.370336 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.370200    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istiod-ca-cert\") pod \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") "
Oct 30 19:16:39.370336 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.370314    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-data" pod "c81ff6d8-e46d-467e-b3fe-49d3e95b34f2" (UID: "c81ff6d8-e46d-467e-b3fe-49d3e95b34f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:39.370449 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.370345    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-workload-socket\") pod \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") "
Oct 30 19:16:39.370449 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.370390    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-podinfo\") pod \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") "
Oct 30 19:16:39.370449 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.370420    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-kube-api-access-n5xvw\" (UniqueName: \"kubernetes.io/projected/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-kube-api-access-n5xvw\") pod \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") "
Oct 30 19:16:39.370607 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.370482    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-workload-certs\") pod \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") "
Oct 30 19:16:39.370607 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.370502    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-credential-socket" pod "c81ff6d8-e46d-467e-b3fe-49d3e95b34f2" (UID: "c81ff6d8-e46d-467e-b3fe-49d3e95b34f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:39.370607 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.370595    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-envoy\") pod \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\" (UID: \"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2\") "
Oct 30 19:16:39.370880 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.370591    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-workload-socket" pod "c81ff6d8-e46d-467e-b3fe-49d3e95b34f2" (UID: "c81ff6d8-e46d-467e-b3fe-49d3e95b34f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:39.371021 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.370641    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istiod-ca-cert" pod "c81ff6d8-e46d-467e-b3fe-49d3e95b34f2" (UID: "c81ff6d8-e46d-467e-b3fe-49d3e95b34f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:39.371021 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.370801    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-workload-certs" pod "c81ff6d8-e46d-467e-b3fe-49d3e95b34f2" (UID: "c81ff6d8-e46d-467e-b3fe-49d3e95b34f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:39.371180 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.371155    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:39.371246 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.371178    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:39.371246 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.371195    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:39.371246 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.371211    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:39.371246 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.371226    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:39.372770 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.372746    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-podinfo" pod "c81ff6d8-e46d-467e-b3fe-49d3e95b34f2" (UID: "c81ff6d8-e46d-467e-b3fe-49d3e95b34f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:16:39.372881 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.372765    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-token" pod "c81ff6d8-e46d-467e-b3fe-49d3e95b34f2" (UID: "c81ff6d8-e46d-467e-b3fe-49d3e95b34f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:39.373145 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.373119    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-envoy" pod "c81ff6d8-e46d-467e-b3fe-49d3e95b34f2" (UID: "c81ff6d8-e46d-467e-b3fe-49d3e95b34f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:39.373259 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.373232    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-kube-api-access-n5xvw" pod "c81ff6d8-e46d-467e-b3fe-49d3e95b34f2" (UID: "c81ff6d8-e46d-467e-b3fe-49d3e95b34f2"). InnerVolumeSpecName "kube-api-access-n5xvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:39.472026 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.471990    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:39.472026 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.472018    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:39.472026 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.472027    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:39.472357 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.472036    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5xvw\" (UniqueName: \"kubernetes.io/projected/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2-kube-api-access-n5xvw\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:39.546318 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.546224    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p" event={"ID":"f7a23dbd-97df-45af-b79b-87ead5172a61","Type":"ContainerStarted","Data":"8a880f53d849a92d172f35f8a62a545d97ea337aaf68457fccfb6f4bd19c3210"}
Oct 30 19:16:39.546318 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.546314    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p" event={"ID":"f7a23dbd-97df-45af-b79b-87ead5172a61","Type":"ContainerStarted","Data":"7bc6ebf821e133b45c119cb08554100b2da5bc433e2cf2871cf8866a81c27be8"}
Oct 30 19:16:39.547655 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.547633    4924 generic.go:356] "Generic (PLEG): container finished" podID="c81ff6d8-e46d-467e-b3fe-49d3e95b34f2" containerID="1912e82c51a53e10f18c7eb541e95ea34ed4712cdad45a9bee99d24ace3a00de" exitCode=0
Oct 30 19:16:39.547738 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.547682    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"
Oct 30 19:16:39.547738 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.547694    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4" event={"ID":"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2","Type":"ContainerDied","Data":"1912e82c51a53e10f18c7eb541e95ea34ed4712cdad45a9bee99d24ace3a00de"}
Oct 30 19:16:39.547738 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.547729    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4" event={"ID":"c81ff6d8-e46d-467e-b3fe-49d3e95b34f2","Type":"ContainerDied","Data":"094d5a80a63a9e7c26b9dc80cab538c6e69baeff5e876b722e263bbb97cf1df1"}
Oct 30 19:16:39.547869 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.547749    4924 scope.go:122] "RemoveContainer" containerID="1912e82c51a53e10f18c7eb541e95ea34ed4712cdad45a9bee99d24ace3a00de"
Oct 30 19:16:39.559857 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.559838    4924 scope.go:122] "RemoveContainer" containerID="1912e82c51a53e10f18c7eb541e95ea34ed4712cdad45a9bee99d24ace3a00de"
Oct 30 19:16:39.560131 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:16:39.560109    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1912e82c51a53e10f18c7eb541e95ea34ed4712cdad45a9bee99d24ace3a00de\": container with ID starting with 1912e82c51a53e10f18c7eb541e95ea34ed4712cdad45a9bee99d24ace3a00de not found: ID does not exist" containerID="1912e82c51a53e10f18c7eb541e95ea34ed4712cdad45a9bee99d24ace3a00de"
Oct 30 19:16:39.560214 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.560138    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1912e82c51a53e10f18c7eb541e95ea34ed4712cdad45a9bee99d24ace3a00de"} err="failed to get container status \"1912e82c51a53e10f18c7eb541e95ea34ed4712cdad45a9bee99d24ace3a00de\": rpc error: code = NotFound desc = could not find container \"1912e82c51a53e10f18c7eb541e95ea34ed4712cdad45a9bee99d24ace3a00de\": container with ID starting with 1912e82c51a53e10f18c7eb541e95ea34ed4712cdad45a9bee99d24ace3a00de not found: ID does not exist"
Oct 30 19:16:39.657911 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.657855    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p" podStartSLOduration=2.6578410740000002 podStartE2EDuration="2.657841074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:39.655913952 +0000 UTC m=+2400.932964396" watchObservedRunningTime="2026-10-30 19:16:39.657841074 +0000 UTC m=+2400.934891519"
Oct 30 19:16:39.766339 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.766307    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"]
Oct 30 19:16:39.770413 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:39.770386    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f80ac-openshift-default-fffccc485-hfzq4"]
Oct 30 19:16:40.559929 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:40.559891    4924 generic.go:356] "Generic (PLEG): container finished" podID="f7a23dbd-97df-45af-b79b-87ead5172a61" containerID="8a880f53d849a92d172f35f8a62a545d97ea337aaf68457fccfb6f4bd19c3210" exitCode=0
Oct 30 19:16:40.560437 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:40.560017    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p" event={"ID":"f7a23dbd-97df-45af-b79b-87ead5172a61","Type":"ContainerDied","Data":"8a880f53d849a92d172f35f8a62a545d97ea337aaf68457fccfb6f4bd19c3210"}
Oct 30 19:16:40.835758 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:40.835677    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c81ff6d8-e46d-467e-b3fe-49d3e95b34f2" path="/var/lib/kubelet/pods/c81ff6d8-e46d-467e-b3fe-49d3e95b34f2/volumes"
Oct 30 19:16:40.999549 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:40.999521    4924 scope.go:122] "RemoveContainer" containerID="44a90434f956799803869114fd1497046f18283a3e4bd433332686613b990c3b"
Oct 30 19:16:41.714698 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:41.714671    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:41.726828 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:41.726807    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p"
Oct 30 19:16:41.826545 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:41.826457    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c" podUID="0580aebc-4e9e-4578-872c-943719ee559f" containerName="payload-processing" containerID="cri-o://5e4b153de2dc19cbf358ec5addca655afdce7fe5f1471a8003e66065b16c1282" gracePeriod=25
Oct 30 19:16:41.903370 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:41.903335    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7a23dbd-97df-45af-b79b-87ead5172a61-kube-api-access-2hg6w\" (UniqueName: \"kubernetes.io/projected/f7a23dbd-97df-45af-b79b-87ead5172a61-kube-api-access-2hg6w\") pod \"f7a23dbd-97df-45af-b79b-87ead5172a61\" (UID: \"f7a23dbd-97df-45af-b79b-87ead5172a61\") "
Oct 30 19:16:41.903588 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:41.903438    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7a23dbd-97df-45af-b79b-87ead5172a61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7a23dbd-97df-45af-b79b-87ead5172a61-maas-api-openshift-ca\") pod \"f7a23dbd-97df-45af-b79b-87ead5172a61\" (UID: \"f7a23dbd-97df-45af-b79b-87ead5172a61\") "
Oct 30 19:16:41.903588 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:41.903512    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7a23dbd-97df-45af-b79b-87ead5172a61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7a23dbd-97df-45af-b79b-87ead5172a61-maas-api-xks-ca\") pod \"f7a23dbd-97df-45af-b79b-87ead5172a61\" (UID: \"f7a23dbd-97df-45af-b79b-87ead5172a61\") "
Oct 30 19:16:41.903863 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:41.903838    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7a23dbd-97df-45af-b79b-87ead5172a61-maas-api-openshift-ca" pod "f7a23dbd-97df-45af-b79b-87ead5172a61" (UID: "f7a23dbd-97df-45af-b79b-87ead5172a61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:41.904056 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:41.904039    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7a23dbd-97df-45af-b79b-87ead5172a61-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:41.905719 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:41.905698    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7a23dbd-97df-45af-b79b-87ead5172a61-maas-api-xks-ca" pod "f7a23dbd-97df-45af-b79b-87ead5172a61" (UID: "f7a23dbd-97df-45af-b79b-87ead5172a61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:41.905908 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:41.905886    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a23dbd-97df-45af-b79b-87ead5172a61-kube-api-access-2hg6w" pod "f7a23dbd-97df-45af-b79b-87ead5172a61" (UID: "f7a23dbd-97df-45af-b79b-87ead5172a61"). InnerVolumeSpecName "kube-api-access-2hg6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:41.928988 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:41.928959    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4" podUID="d3dd0455-39e6-4b1b-80ee-ffbde02d4f00" containerName="payload-pre-processing" containerID="cri-o://389d884c1ef4feb0839e774bb9640ea12d9d699110005bcda5dda8022a41cf29" gracePeriod=25
Oct 30 19:16:42.005259 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.005224    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hg6w\" (UniqueName: \"kubernetes.io/projected/f7a23dbd-97df-45af-b79b-87ead5172a61-kube-api-access-2hg6w\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:42.005259 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.005260    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7a23dbd-97df-45af-b79b-87ead5172a61-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:42.041511 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.041476    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:42.299234 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.299210    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:16:42.379025 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.378990    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:42.410200 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.410132    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-workload-socket\") pod \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") "
Oct 30 19:16:42.410200 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.410192    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-workload-certs\") pod \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") "
Oct 30 19:16:42.410374 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.410335    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-data\") pod \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") "
Oct 30 19:16:42.410423 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.410394    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istiod-ca-cert\") pod \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") "
Oct 30 19:16:42.410469 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.410450    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-workload-socket" pod "0c9fa098-37e6-48ad-ba03-1879f9017f7c" (UID: "0c9fa098-37e6-48ad-ba03-1879f9017f7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:42.410511 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.410454    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-podinfo\") pod \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") "
Oct 30 19:16:42.410550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.410508    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-workload-certs" pod "0c9fa098-37e6-48ad-ba03-1879f9017f7c" (UID: "0c9fa098-37e6-48ad-ba03-1879f9017f7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:42.410550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.410519    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-envoy\") pod \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") "
Oct 30 19:16:42.410639 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.410620    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c9fa098-37e6-48ad-ba03-1879f9017f7c-kube-api-access-7mzrs\" (UniqueName: \"kubernetes.io/projected/0c9fa098-37e6-48ad-ba03-1879f9017f7c-kube-api-access-7mzrs\") pod \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") "
Oct 30 19:16:42.410682 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.410624    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-data" pod "0c9fa098-37e6-48ad-ba03-1879f9017f7c" (UID: "0c9fa098-37e6-48ad-ba03-1879f9017f7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:42.410720 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.410689    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-token\" (UniqueName: \"kubernetes.io/projected/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-token\") pod \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") "
Oct 30 19:16:42.410769 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.410717    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istiod-ca-cert" pod "0c9fa098-37e6-48ad-ba03-1879f9017f7c" (UID: "0c9fa098-37e6-48ad-ba03-1879f9017f7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:42.410769 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.410735    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-credential-socket\") pod \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\" (UID: \"0c9fa098-37e6-48ad-ba03-1879f9017f7c\") "
Oct 30 19:16:42.411112 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.411054    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-credential-socket" pod "0c9fa098-37e6-48ad-ba03-1879f9017f7c" (UID: "0c9fa098-37e6-48ad-ba03-1879f9017f7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:42.411468 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.411344    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:42.411468 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.411368    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:42.411468 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.411381    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:42.411468 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.411393    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:42.411468 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.411406    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:42.413298 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.413272    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-envoy" pod "0c9fa098-37e6-48ad-ba03-1879f9017f7c" (UID: "0c9fa098-37e6-48ad-ba03-1879f9017f7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:42.413408 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.413341    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-podinfo" pod "0c9fa098-37e6-48ad-ba03-1879f9017f7c" (UID: "0c9fa098-37e6-48ad-ba03-1879f9017f7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:16:42.413552 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.413533    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9fa098-37e6-48ad-ba03-1879f9017f7c-kube-api-access-7mzrs" pod "0c9fa098-37e6-48ad-ba03-1879f9017f7c" (UID: "0c9fa098-37e6-48ad-ba03-1879f9017f7c"). InnerVolumeSpecName "kube-api-access-7mzrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:42.413807 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.413781    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-token" pod "0c9fa098-37e6-48ad-ba03-1879f9017f7c" (UID: "0c9fa098-37e6-48ad-ba03-1879f9017f7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:42.512280 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.512239    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:42.512280 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.512273    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:42.512610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.512289    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mzrs\" (UniqueName: \"kubernetes.io/projected/0c9fa098-37e6-48ad-ba03-1879f9017f7c-kube-api-access-7mzrs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:42.512610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.512304    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9fa098-37e6-48ad-ba03-1879f9017f7c-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:42.576193 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.576159    4924 generic.go:356] "Generic (PLEG): container finished" podID="0c9fa098-37e6-48ad-ba03-1879f9017f7c" containerID="22b4c7af931de94782cbf1bde654e507b4be99e1a07af3ca9a39baa355e46197" exitCode=0
Oct 30 19:16:42.576403 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.576208    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"
Oct 30 19:16:42.576403 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.576254    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf" event={"ID":"0c9fa098-37e6-48ad-ba03-1879f9017f7c","Type":"ContainerDied","Data":"22b4c7af931de94782cbf1bde654e507b4be99e1a07af3ca9a39baa355e46197"}
Oct 30 19:16:42.576403 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.576319    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf" event={"ID":"0c9fa098-37e6-48ad-ba03-1879f9017f7c","Type":"ContainerDied","Data":"a21e4af3ba878880be39a86ad637dc28d088f1257e908d7fde637cd3571db1f0"}
Oct 30 19:16:42.576403 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.576351    4924 scope.go:122] "RemoveContainer" containerID="22b4c7af931de94782cbf1bde654e507b4be99e1a07af3ca9a39baa355e46197"
Oct 30 19:16:42.577891 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.577868    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p"
Oct 30 19:16:42.578015 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.577903    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p" event={"ID":"f7a23dbd-97df-45af-b79b-87ead5172a61","Type":"ContainerDied","Data":"7bc6ebf821e133b45c119cb08554100b2da5bc433e2cf2871cf8866a81c27be8"}
Oct 30 19:16:42.578015 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.577951    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bc6ebf821e133b45c119cb08554100b2da5bc433e2cf2871cf8866a81c27be8"
Oct 30 19:16:42.590971 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.590952    4924 scope.go:122] "RemoveContainer" containerID="a2461b074d3d188c16e488a0dc09b2941276e4a70b34af07f3fd43c345633329"
Oct 30 19:16:42.600972 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.600954    4924 scope.go:122] "RemoveContainer" containerID="22b4c7af931de94782cbf1bde654e507b4be99e1a07af3ca9a39baa355e46197"
Oct 30 19:16:42.601280 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:16:42.601261    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b4c7af931de94782cbf1bde654e507b4be99e1a07af3ca9a39baa355e46197\": container with ID starting with 22b4c7af931de94782cbf1bde654e507b4be99e1a07af3ca9a39baa355e46197 not found: ID does not exist" containerID="22b4c7af931de94782cbf1bde654e507b4be99e1a07af3ca9a39baa355e46197"
Oct 30 19:16:42.601336 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.601287    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b4c7af931de94782cbf1bde654e507b4be99e1a07af3ca9a39baa355e46197"} err="failed to get container status \"22b4c7af931de94782cbf1bde654e507b4be99e1a07af3ca9a39baa355e46197\": rpc error: code = NotFound desc = could not find container \"22b4c7af931de94782cbf1bde654e507b4be99e1a07af3ca9a39baa355e46197\": container with ID starting with 22b4c7af931de94782cbf1bde654e507b4be99e1a07af3ca9a39baa355e46197 not found: ID does not exist"
Oct 30 19:16:42.601336 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.601304    4924 scope.go:122] "RemoveContainer" containerID="a2461b074d3d188c16e488a0dc09b2941276e4a70b34af07f3fd43c345633329"
Oct 30 19:16:42.601578 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:16:42.601551    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2461b074d3d188c16e488a0dc09b2941276e4a70b34af07f3fd43c345633329\": container with ID starting with a2461b074d3d188c16e488a0dc09b2941276e4a70b34af07f3fd43c345633329 not found: ID does not exist" containerID="a2461b074d3d188c16e488a0dc09b2941276e4a70b34af07f3fd43c345633329"
Oct 30 19:16:42.601619 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.601587    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2461b074d3d188c16e488a0dc09b2941276e4a70b34af07f3fd43c345633329"} err="failed to get container status \"a2461b074d3d188c16e488a0dc09b2941276e4a70b34af07f3fd43c345633329\": rpc error: code = NotFound desc = could not find container \"a2461b074d3d188c16e488a0dc09b2941276e4a70b34af07f3fd43c345633329\": container with ID starting with a2461b074d3d188c16e488a0dc09b2941276e4a70b34af07f3fd43c345633329 not found: ID does not exist"
Oct 30 19:16:42.660399 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.660328    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"]
Oct 30 19:16:42.666542 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.666515    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"]
Oct 30 19:16:42.680702 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.680680    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"]
Oct 30 19:16:42.740193 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.740163    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"]
Oct 30 19:16:42.750750 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.750695    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:42.755045 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.755017    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f3a0e7-openshift-default-c5bff4799-2lztf"]
Oct 30 19:16:42.834405 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:42.834370    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c9fa098-37e6-48ad-ba03-1879f9017f7c" path="/var/lib/kubelet/pods/0c9fa098-37e6-48ad-ba03-1879f9017f7c/volumes"
Oct 30 19:16:43.072038 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:43.071951    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:43.426167 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:43.426135    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:43.749686 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:43.749606    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:44.109531 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:44.109500    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:44.426235 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:44.426095    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:44.499255 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:44.499129    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c" podUID="0580aebc-4e9e-4578-872c-943719ee559f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:16:44.504364 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:44.504320    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4" podUID="d3dd0455-39e6-4b1b-80ee-ffbde02d4f00" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:16:44.791057 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:44.790978    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:44.799953 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:44.799916    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5" podUID="925a4de9-b451-4263-9a1c-eebbbae4a08c" containerName="main" containerID="cri-o://eb96402c7314a0b89230f8b32897eb32a3f9e812c752e09395880f680cdb5aa3" gracePeriod=45
Oct 30 19:16:44.881142 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:44.881099    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5" podUID="925a4de9-b451-4263-9a1c-eebbbae4a08c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": dial tcp 10.132.0.63:8000: connect: connection refused"
Oct 30 19:16:45.071519 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.071493    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 19:16:45.140088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.140050    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/925a4de9-b451-4263-9a1c-eebbbae4a08c-tls-certs\" (UniqueName: \"kubernetes.io/secret/925a4de9-b451-4263-9a1c-eebbbae4a08c-tls-certs\") pod \"925a4de9-b451-4263-9a1c-eebbbae4a08c\" (UID: \"925a4de9-b451-4263-9a1c-eebbbae4a08c\") "
Oct 30 19:16:45.140274 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.140107    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-model-cache\") pod \"925a4de9-b451-4263-9a1c-eebbbae4a08c\" (UID: \"925a4de9-b451-4263-9a1c-eebbbae4a08c\") "
Oct 30 19:16:45.140274 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.140155    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-tmp-dir\") pod \"925a4de9-b451-4263-9a1c-eebbbae4a08c\" (UID: \"925a4de9-b451-4263-9a1c-eebbbae4a08c\") "
Oct 30 19:16:45.140274 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.140180    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-home\" (UniqueName: \"kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-home\") pod \"925a4de9-b451-4263-9a1c-eebbbae4a08c\" (UID: \"925a4de9-b451-4263-9a1c-eebbbae4a08c\") "
Oct 30 19:16:45.140274 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.140209    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-dshm\") pod \"925a4de9-b451-4263-9a1c-eebbbae4a08c\" (UID: \"925a4de9-b451-4263-9a1c-eebbbae4a08c\") "
Oct 30 19:16:45.140274 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.140273    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/925a4de9-b451-4263-9a1c-eebbbae4a08c-kube-api-access-8mjzd\" (UniqueName: \"kubernetes.io/projected/925a4de9-b451-4263-9a1c-eebbbae4a08c-kube-api-access-8mjzd\") pod \"925a4de9-b451-4263-9a1c-eebbbae4a08c\" (UID: \"925a4de9-b451-4263-9a1c-eebbbae4a08c\") "
Oct 30 19:16:45.140571 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.140443    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-home" pod "925a4de9-b451-4263-9a1c-eebbbae4a08c" (UID: "925a4de9-b451-4263-9a1c-eebbbae4a08c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:45.140571 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.140450    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-model-cache" pod "925a4de9-b451-4263-9a1c-eebbbae4a08c" (UID: "925a4de9-b451-4263-9a1c-eebbbae4a08c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:45.140571 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.140466    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-tmp-dir" pod "925a4de9-b451-4263-9a1c-eebbbae4a08c" (UID: "925a4de9-b451-4263-9a1c-eebbbae4a08c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:45.140737 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.140656    4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-model-cache\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:45.140737 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.140670    4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-tmp-dir\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:45.140737 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.140679    4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-home\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:45.142487 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.142463    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-dshm" pod "925a4de9-b451-4263-9a1c-eebbbae4a08c" (UID: "925a4de9-b451-4263-9a1c-eebbbae4a08c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:45.142601 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.142496    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925a4de9-b451-4263-9a1c-eebbbae4a08c-kube-api-access-8mjzd" pod "925a4de9-b451-4263-9a1c-eebbbae4a08c" (UID: "925a4de9-b451-4263-9a1c-eebbbae4a08c"). InnerVolumeSpecName "kube-api-access-8mjzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:45.142601 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.142554    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/925a4de9-b451-4263-9a1c-eebbbae4a08c-tls-certs" pod "925a4de9-b451-4263-9a1c-eebbbae4a08c" (UID: "925a4de9-b451-4263-9a1c-eebbbae4a08c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:45.242592 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.242552    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mjzd\" (UniqueName: \"kubernetes.io/projected/925a4de9-b451-4263-9a1c-eebbbae4a08c-kube-api-access-8mjzd\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:45.242592 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.242588    4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/925a4de9-b451-4263-9a1c-eebbbae4a08c-tls-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:45.242859 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.242603    4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/925a4de9-b451-4263-9a1c-eebbbae4a08c-dshm\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:45.601051 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.601012    4924 generic.go:356] "Generic (PLEG): container finished" podID="925a4de9-b451-4263-9a1c-eebbbae4a08c" containerID="eb96402c7314a0b89230f8b32897eb32a3f9e812c752e09395880f680cdb5aa3" exitCode=0
Oct 30 19:16:45.601279 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.601057    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"
Oct 30 19:16:45.601279 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.601106    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5" event={"ID":"925a4de9-b451-4263-9a1c-eebbbae4a08c","Type":"ContainerDied","Data":"eb96402c7314a0b89230f8b32897eb32a3f9e812c752e09395880f680cdb5aa3"}
Oct 30 19:16:45.601279 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.601151    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5" event={"ID":"925a4de9-b451-4263-9a1c-eebbbae4a08c","Type":"ContainerDied","Data":"8aabbbedd9a6301d65e391c9428b3b371dd3934768f9179ecc58901072ae602a"}
Oct 30 19:16:45.601279 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.601174    4924 scope.go:122] "RemoveContainer" containerID="eb96402c7314a0b89230f8b32897eb32a3f9e812c752e09395880f680cdb5aa3"
Oct 30 19:16:45.611487 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.611470    4924 scope.go:122] "RemoveContainer" containerID="eb96402c7314a0b89230f8b32897eb32a3f9e812c752e09395880f680cdb5aa3"
Oct 30 19:16:45.611788 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:16:45.611764    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb96402c7314a0b89230f8b32897eb32a3f9e812c752e09395880f680cdb5aa3\": container with ID starting with eb96402c7314a0b89230f8b32897eb32a3f9e812c752e09395880f680cdb5aa3 not found: ID does not exist" containerID="eb96402c7314a0b89230f8b32897eb32a3f9e812c752e09395880f680cdb5aa3"
Oct 30 19:16:45.611852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.611800    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb96402c7314a0b89230f8b32897eb32a3f9e812c752e09395880f680cdb5aa3"} err="failed to get container status \"eb96402c7314a0b89230f8b32897eb32a3f9e812c752e09395880f680cdb5aa3\": rpc error: code = NotFound desc = could not find container \"eb96402c7314a0b89230f8b32897eb32a3f9e812c752e09395880f680cdb5aa3\": container with ID starting with eb96402c7314a0b89230f8b32897eb32a3f9e812c752e09395880f680cdb5aa3 not found: ID does not exist"
Oct 30 19:16:45.711456 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:16:45.711422    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:16:45.775511 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.775476    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"]
Oct 30 19:16:45.779998 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:45.779971    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-75e91b/simulator-w4-75e91b-kserve-5f668ff968-kpwc5"]
Oct 30 19:16:46.066057 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:46.065977    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:46.835152 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:46.835117    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="925a4de9-b451-4263-9a1c-eebbbae4a08c" path="/var/lib/kubelet/pods/925a4de9-b451-4263-9a1c-eebbbae4a08c/volumes"
Oct 30 19:16:49.400655 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:49.400617    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d"]
Oct 30 19:16:49.500549 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:49.500516    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf"]
Oct 30 19:16:49.781943 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:49.781857    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:50.096293 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:50.096260    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:51.527400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:51.527358    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:54.147353 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.147321    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baeda332-g5dvx"]
Oct 30 19:16:54.148048 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.148026    4924 cpu_manager.go:414] "containerMap: removing container" podUID="c81ff6d8-e46d-467e-b3fe-49d3e95b34f2" containerName="istio-proxy"
Oct 30 19:16:54.148202 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.148050    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c81ff6d8-e46d-467e-b3fe-49d3e95b34f2" containerName="istio-proxy"
Oct 30 19:16:54.148202 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.148094    4924 cpu_manager.go:414] "containerMap: removing container" podUID="0c9fa098-37e6-48ad-ba03-1879f9017f7c" containerName="istio-proxy"
Oct 30 19:16:54.148202 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.148103    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9fa098-37e6-48ad-ba03-1879f9017f7c" containerName="istio-proxy"
Oct 30 19:16:54.148202 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.148118    4924 cpu_manager.go:414] "containerMap: removing container" podUID="34498bbd-c6eb-48b9-8003-d863dd955292" containerName="istio-proxy"
Oct 30 19:16:54.148202 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.148127    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34498bbd-c6eb-48b9-8003-d863dd955292" containerName="istio-proxy"
Oct 30 19:16:54.148202 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.148140    4924 cpu_manager.go:414] "containerMap: removing container" podUID="f7a23dbd-97df-45af-b79b-87ead5172a61" containerName="revoke-keys"
Oct 30 19:16:54.148202 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.148147    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7a23dbd-97df-45af-b79b-87ead5172a61" containerName="revoke-keys"
Oct 30 19:16:54.148563 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.148206    4924 cpu_manager.go:414] "containerMap: removing container" podUID="925a4de9-b451-4263-9a1c-eebbbae4a08c" containerName="main"
Oct 30 19:16:54.148563 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.148214    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="925a4de9-b451-4263-9a1c-eebbbae4a08c" containerName="main"
Oct 30 19:16:54.148563 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.148430    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="925a4de9-b451-4263-9a1c-eebbbae4a08c" containerName="main"
Oct 30 19:16:54.148563 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.148449    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="c81ff6d8-e46d-467e-b3fe-49d3e95b34f2" containerName="istio-proxy"
Oct 30 19:16:54.148563 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.148461    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9fa098-37e6-48ad-ba03-1879f9017f7c" containerName="istio-proxy"
Oct 30 19:16:54.148563 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.148469    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="34498bbd-c6eb-48b9-8003-d863dd955292" containerName="istio-proxy"
Oct 30 19:16:54.148563 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.148477    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7a23dbd-97df-45af-b79b-87ead5172a61" containerName="revoke-keys"
Oct 30 19:16:54.153535 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.153513    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeda332-g5dvx"
Oct 30 19:16:54.201781 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.201756    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7da0a81c-92hnt"]
Oct 30 19:16:54.202410 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.202387    4924 cpu_manager.go:414] "containerMap: removing container" podUID="0c9fa098-37e6-48ad-ba03-1879f9017f7c" containerName="istio-proxy"
Oct 30 19:16:54.202572 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.202555    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9fa098-37e6-48ad-ba03-1879f9017f7c" containerName="istio-proxy"
Oct 30 19:16:54.202977 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.202935    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9fa098-37e6-48ad-ba03-1879f9017f7c" containerName="istio-proxy"
Oct 30 19:16:54.206693 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.206676    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baeda332-g5dvx"]
Oct 30 19:16:54.206783 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.206758    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da0a81c-92hnt"
Oct 30 19:16:54.245188 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.245158    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:16:54.245372 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.245204    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:16:54.247251 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.247229    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:16:54.247314 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.247275    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:16:54.255045 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.255022    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37d1db7a-108c-4972-8223-3ed0a626a6dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7da0a81c-92hnt\" (UID: \"37d1db7a-108c-4972-8223-3ed0a626a6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da0a81c-92hnt"
Oct 30 19:16:54.255233 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.255211    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37d1db7a-108c-4972-8223-3ed0a626a6dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7da0a81c-92hnt\" (UID: \"37d1db7a-108c-4972-8223-3ed0a626a6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da0a81c-92hnt"
Oct 30 19:16:54.255334 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.255316    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjvsd\" (UniqueName: \"kubernetes.io/projected/fb3461be-6387-4cf8-8d71-082737fdd0d8-kube-api-access-qjvsd\") pod \"maas-api-revoke-sub-baeda332-g5dvx\" (UID: \"fb3461be-6387-4cf8-8d71-082737fdd0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeda332-g5dvx"
Oct 30 19:16:54.255379 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.255351    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmx29\" (UniqueName: \"kubernetes.io/projected/37d1db7a-108c-4972-8223-3ed0a626a6dc-kube-api-access-tmx29\") pod \"maas-api-revoke-sub-7da0a81c-92hnt\" (UID: \"37d1db7a-108c-4972-8223-3ed0a626a6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da0a81c-92hnt"
Oct 30 19:16:54.255379 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.255372    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb3461be-6387-4cf8-8d71-082737fdd0d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baeda332-g5dvx\" (UID: \"fb3461be-6387-4cf8-8d71-082737fdd0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeda332-g5dvx"
Oct 30 19:16:54.255482 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.255466    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb3461be-6387-4cf8-8d71-082737fdd0d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baeda332-g5dvx\" (UID: \"fb3461be-6387-4cf8-8d71-082737fdd0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeda332-g5dvx"
Oct 30 19:16:54.265994 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.265965    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7da0a81c-92hnt"]
Oct 30 19:16:54.356844 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.356796    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb3461be-6387-4cf8-8d71-082737fdd0d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baeda332-g5dvx\" (UID: \"fb3461be-6387-4cf8-8d71-082737fdd0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeda332-g5dvx"
Oct 30 19:16:54.357035 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.356890    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37d1db7a-108c-4972-8223-3ed0a626a6dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7da0a81c-92hnt\" (UID: \"37d1db7a-108c-4972-8223-3ed0a626a6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da0a81c-92hnt"
Oct 30 19:16:54.357035 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.357005    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37d1db7a-108c-4972-8223-3ed0a626a6dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7da0a81c-92hnt\" (UID: \"37d1db7a-108c-4972-8223-3ed0a626a6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da0a81c-92hnt"
Oct 30 19:16:54.357165 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.357135    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjvsd\" (UniqueName: \"kubernetes.io/projected/fb3461be-6387-4cf8-8d71-082737fdd0d8-kube-api-access-qjvsd\") pod \"maas-api-revoke-sub-baeda332-g5dvx\" (UID: \"fb3461be-6387-4cf8-8d71-082737fdd0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeda332-g5dvx"
Oct 30 19:16:54.357222 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.357167    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmx29\" (UniqueName: \"kubernetes.io/projected/37d1db7a-108c-4972-8223-3ed0a626a6dc-kube-api-access-tmx29\") pod \"maas-api-revoke-sub-7da0a81c-92hnt\" (UID: \"37d1db7a-108c-4972-8223-3ed0a626a6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da0a81c-92hnt"
Oct 30 19:16:54.357222 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.357196    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb3461be-6387-4cf8-8d71-082737fdd0d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baeda332-g5dvx\" (UID: \"fb3461be-6387-4cf8-8d71-082737fdd0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeda332-g5dvx"
Oct 30 19:16:54.357689 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.357664    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37d1db7a-108c-4972-8223-3ed0a626a6dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7da0a81c-92hnt\" (UID: \"37d1db7a-108c-4972-8223-3ed0a626a6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da0a81c-92hnt"
Oct 30 19:16:54.357853 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.357831    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb3461be-6387-4cf8-8d71-082737fdd0d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baeda332-g5dvx\" (UID: \"fb3461be-6387-4cf8-8d71-082737fdd0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeda332-g5dvx"
Oct 30 19:16:54.359410 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.359388    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb3461be-6387-4cf8-8d71-082737fdd0d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baeda332-g5dvx\" (UID: \"fb3461be-6387-4cf8-8d71-082737fdd0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeda332-g5dvx"
Oct 30 19:16:54.359522 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.359408    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37d1db7a-108c-4972-8223-3ed0a626a6dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7da0a81c-92hnt\" (UID: \"37d1db7a-108c-4972-8223-3ed0a626a6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da0a81c-92hnt"
Oct 30 19:16:54.401364 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.401263    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d" podUID="b9cd1a58-ecd4-4611-aaac-28109cd2981a" containerName="payload-processing" containerID="cri-o://2164deca5b6f9cbd971790debf5d890e560ef56d52597679275771209c898bc1" gracePeriod=25
Oct 30 19:16:54.414083 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.414044    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjvsd\" (UniqueName: \"kubernetes.io/projected/fb3461be-6387-4cf8-8d71-082737fdd0d8-kube-api-access-qjvsd\") pod \"maas-api-revoke-sub-baeda332-g5dvx\" (UID: \"fb3461be-6387-4cf8-8d71-082737fdd0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeda332-g5dvx"
Oct 30 19:16:54.414232 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.414211    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmx29\" (UniqueName: \"kubernetes.io/projected/37d1db7a-108c-4972-8223-3ed0a626a6dc-kube-api-access-tmx29\") pod \"maas-api-revoke-sub-7da0a81c-92hnt\" (UID: \"37d1db7a-108c-4972-8223-3ed0a626a6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da0a81c-92hnt"
Oct 30 19:16:54.467332 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.467301    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeda332-g5dvx"
Oct 30 19:16:54.494710 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.494675    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c" podUID="0580aebc-4e9e-4578-872c-943719ee559f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:16:54.494833 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.494758    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4" podUID="d3dd0455-39e6-4b1b-80ee-ffbde02d4f00" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:16:54.501395 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.501363    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf" podUID="dc3baffc-49c4-4101-9bf4-047229d89000" containerName="payload-pre-processing" containerID="cri-o://922c79e8290ed4f53737c6383a853630095dcff3ea014c272638e4fdece41ef1" gracePeriod=25
Oct 30 19:16:54.524810 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.524787    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da0a81c-92hnt"
Oct 30 19:16:54.835973 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.835942    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baeda332-g5dvx"]
Oct 30 19:16:54.837266 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:16:54.837237    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb3461be_6387_4cf8_8d71_082737fdd0d8.slice/crio-4a6eae6577bd1db2ded58d7a10dfb569b0c9baef2a37ea55cea0233762a93d10 WatchSource:0}: Error finding container 4a6eae6577bd1db2ded58d7a10dfb569b0c9baef2a37ea55cea0233762a93d10: Status 404 returned error can't find the container with id 4a6eae6577bd1db2ded58d7a10dfb569b0c9baef2a37ea55cea0233762a93d10
Oct 30 19:16:54.938874 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:54.938849    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7da0a81c-92hnt"]
Oct 30 19:16:55.008508 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:16:55.008475    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d1db7a_108c_4972_8223_3ed0a626a6dc.slice/crio-a6757af70c65d29b1a2b3aec7918725e9ed0a057e75fb355b34a8239115cab98 WatchSource:0}: Error finding container a6757af70c65d29b1a2b3aec7918725e9ed0a057e75fb355b34a8239115cab98: Status 404 returned error can't find the container with id a6757af70c65d29b1a2b3aec7918725e9ed0a057e75fb355b34a8239115cab98
Oct 30 19:16:55.064413 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:55.064385    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:16:55.684830 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:55.684786    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da0a81c-92hnt" event={"ID":"37d1db7a-108c-4972-8223-3ed0a626a6dc","Type":"ContainerStarted","Data":"669404bc4632cca1cc09bbd78305d74c03759516a8e8ff235069e45e44f59acf"}
Oct 30 19:16:55.685293 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:55.684853    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da0a81c-92hnt" event={"ID":"37d1db7a-108c-4972-8223-3ed0a626a6dc","Type":"ContainerStarted","Data":"a6757af70c65d29b1a2b3aec7918725e9ed0a057e75fb355b34a8239115cab98"}
Oct 30 19:16:55.686610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:55.686588    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeda332-g5dvx" event={"ID":"fb3461be-6387-4cf8-8d71-082737fdd0d8","Type":"ContainerStarted","Data":"2dc671f260bd82006191b6aa46c5a218e861b55272cb7b922ba4447bd2870970"}
Oct 30 19:16:55.686720 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:55.686627    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeda332-g5dvx" event={"ID":"fb3461be-6387-4cf8-8d71-082737fdd0d8","Type":"ContainerStarted","Data":"4a6eae6577bd1db2ded58d7a10dfb569b0c9baef2a37ea55cea0233762a93d10"}
Oct 30 19:16:55.794949 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:55.794893    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da0a81c-92hnt" podStartSLOduration=1.7948806880000001 podStartE2EDuration="1.794880688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:55.792627958 +0000 UTC m=+2417.069678402" watchObservedRunningTime="2026-10-30 19:16:55.794880688 +0000 UTC m=+2417.071931134"
Oct 30 19:16:55.901324 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:55.901272    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeda332-g5dvx" podStartSLOduration=1.901258374 podStartE2EDuration="1.901258374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:55.900158001 +0000 UTC m=+2417.177208442" watchObservedRunningTime="2026-10-30 19:16:55.901258374 +0000 UTC m=+2417.178308826"
Oct 30 19:16:56.696000 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:56.695968    4924 generic.go:356] "Generic (PLEG): container finished" podID="37d1db7a-108c-4972-8223-3ed0a626a6dc" containerID="669404bc4632cca1cc09bbd78305d74c03759516a8e8ff235069e45e44f59acf" exitCode=0
Oct 30 19:16:56.696418 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:56.696047    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da0a81c-92hnt" event={"ID":"37d1db7a-108c-4972-8223-3ed0a626a6dc","Type":"ContainerDied","Data":"669404bc4632cca1cc09bbd78305d74c03759516a8e8ff235069e45e44f59acf"}
Oct 30 19:16:56.697634 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:56.697610    4924 generic.go:356] "Generic (PLEG): container finished" podID="fb3461be-6387-4cf8-8d71-082737fdd0d8" containerID="2dc671f260bd82006191b6aa46c5a218e861b55272cb7b922ba4447bd2870970" exitCode=0
Oct 30 19:16:56.697765 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:56.697665    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeda332-g5dvx" event={"ID":"fb3461be-6387-4cf8-8d71-082737fdd0d8","Type":"ContainerDied","Data":"2dc671f260bd82006191b6aa46c5a218e861b55272cb7b922ba4447bd2870970"}
Oct 30 19:16:57.682025 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:57.681983    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj" podUID="51f3e639-9cd4-4aaa-a568-f19b6d2704fd" containerName="main" containerID="cri-o://aa0f8e048bb1d418eac1c5eb0d71205e7830315a6f63f556312a1f9aa6c504fa" gracePeriod=45
Oct 30 19:16:57.686663 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:57.686635    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86" podUID="60a58597-fdc4-4722-8d2a-29e475f84a76" containerName="main" containerID="cri-o://480ca2e8a1a174a796513c83a2316f1e00d053c13352d0c7790b03d7aabd619a" gracePeriod=45
Oct 30 19:16:57.700087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:57.700030    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2" podUID="fdf4711c-418a-436b-848a-d46974566573" containerName="main" containerID="cri-o://6a72e506dbbd782b6d51df437ebcbfd0e90ccd639153874a83f28c29827d33ad" gracePeriod=45
Oct 30 19:16:57.756166 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:57.756121    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj" podUID="51f3e639-9cd4-4aaa-a568-f19b6d2704fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": dial tcp 10.132.0.92:8000: connect: connection refused"
Oct 30 19:16:57.899323 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:57.897914    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2" podUID="fdf4711c-418a-436b-848a-d46974566573" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": dial tcp 10.132.0.61:8000: connect: connection refused"
Oct 30 19:16:57.928735 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:57.928705    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da0a81c-92hnt"
Oct 30 19:16:57.932313 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:57.932237    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeda332-g5dvx"
Oct 30 19:16:58.014405 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.014374    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d1db7a-108c-4972-8223-3ed0a626a6dc-kube-api-access-tmx29\" (UniqueName: \"kubernetes.io/projected/37d1db7a-108c-4972-8223-3ed0a626a6dc-kube-api-access-tmx29\") pod \"37d1db7a-108c-4972-8223-3ed0a626a6dc\" (UID: \"37d1db7a-108c-4972-8223-3ed0a626a6dc\") "
Oct 30 19:16:58.014601 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.014463    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb3461be-6387-4cf8-8d71-082737fdd0d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb3461be-6387-4cf8-8d71-082737fdd0d8-maas-api-xks-ca\") pod \"fb3461be-6387-4cf8-8d71-082737fdd0d8\" (UID: \"fb3461be-6387-4cf8-8d71-082737fdd0d8\") "
Oct 30 19:16:58.014601 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.014495    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb3461be-6387-4cf8-8d71-082737fdd0d8-kube-api-access-qjvsd\" (UniqueName: \"kubernetes.io/projected/fb3461be-6387-4cf8-8d71-082737fdd0d8-kube-api-access-qjvsd\") pod \"fb3461be-6387-4cf8-8d71-082737fdd0d8\" (UID: \"fb3461be-6387-4cf8-8d71-082737fdd0d8\") "
Oct 30 19:16:58.014601 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.014565    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37d1db7a-108c-4972-8223-3ed0a626a6dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37d1db7a-108c-4972-8223-3ed0a626a6dc-maas-api-openshift-ca\") pod \"37d1db7a-108c-4972-8223-3ed0a626a6dc\" (UID: \"37d1db7a-108c-4972-8223-3ed0a626a6dc\") "
Oct 30 19:16:58.014825 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.014665    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37d1db7a-108c-4972-8223-3ed0a626a6dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37d1db7a-108c-4972-8223-3ed0a626a6dc-maas-api-xks-ca\") pod \"37d1db7a-108c-4972-8223-3ed0a626a6dc\" (UID: \"37d1db7a-108c-4972-8223-3ed0a626a6dc\") "
Oct 30 19:16:58.014825 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.014726    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb3461be-6387-4cf8-8d71-082737fdd0d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb3461be-6387-4cf8-8d71-082737fdd0d8-maas-api-openshift-ca\") pod \"fb3461be-6387-4cf8-8d71-082737fdd0d8\" (UID: \"fb3461be-6387-4cf8-8d71-082737fdd0d8\") "
Oct 30 19:16:58.015954 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.015919    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d1db7a-108c-4972-8223-3ed0a626a6dc-maas-api-openshift-ca" pod "37d1db7a-108c-4972-8223-3ed0a626a6dc" (UID: "37d1db7a-108c-4972-8223-3ed0a626a6dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:58.016258 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.016224    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb3461be-6387-4cf8-8d71-082737fdd0d8-maas-api-openshift-ca" pod "fb3461be-6387-4cf8-8d71-082737fdd0d8" (UID: "fb3461be-6387-4cf8-8d71-082737fdd0d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:16:58.025639 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.025580    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d1db7a-108c-4972-8223-3ed0a626a6dc-kube-api-access-tmx29" pod "37d1db7a-108c-4972-8223-3ed0a626a6dc" (UID: "37d1db7a-108c-4972-8223-3ed0a626a6dc"). InnerVolumeSpecName "kube-api-access-tmx29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:58.026029 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.025986    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb3461be-6387-4cf8-8d71-082737fdd0d8-maas-api-xks-ca" pod "fb3461be-6387-4cf8-8d71-082737fdd0d8" (UID: "fb3461be-6387-4cf8-8d71-082737fdd0d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:58.033174 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.033098    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb3461be-6387-4cf8-8d71-082737fdd0d8-kube-api-access-qjvsd" pod "fb3461be-6387-4cf8-8d71-082737fdd0d8" (UID: "fb3461be-6387-4cf8-8d71-082737fdd0d8"). InnerVolumeSpecName "kube-api-access-qjvsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:58.033523 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.033490    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37d1db7a-108c-4972-8223-3ed0a626a6dc-maas-api-xks-ca" pod "37d1db7a-108c-4972-8223-3ed0a626a6dc" (UID: "37d1db7a-108c-4972-8223-3ed0a626a6dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:58.047700 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.047672    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 19:16:58.049766 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.049741    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 19:16:58.053789 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.053765    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 19:16:58.116286 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.116257    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60a58597-fdc4-4722-8d2a-29e475f84a76-kube-api-access-5s4bl\" (UniqueName: \"kubernetes.io/projected/60a58597-fdc4-4722-8d2a-29e475f84a76-kube-api-access-5s4bl\") pod \"60a58597-fdc4-4722-8d2a-29e475f84a76\" (UID: \"60a58597-fdc4-4722-8d2a-29e475f84a76\") "
Oct 30 19:16:58.116465 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.116306    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdf4711c-418a-436b-848a-d46974566573-kube-api-access-tvm88\" (UniqueName: \"kubernetes.io/projected/fdf4711c-418a-436b-848a-d46974566573-kube-api-access-tvm88\") pod \"fdf4711c-418a-436b-848a-d46974566573\" (UID: \"fdf4711c-418a-436b-848a-d46974566573\") "
Oct 30 19:16:58.116465 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.116361    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-dshm\") pod \"fdf4711c-418a-436b-848a-d46974566573\" (UID: \"fdf4711c-418a-436b-848a-d46974566573\") "
Oct 30 19:16:58.116465 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.116402    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdf4711c-418a-436b-848a-d46974566573-tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf4711c-418a-436b-848a-d46974566573-tls-certs\") pod \"fdf4711c-418a-436b-848a-d46974566573\" (UID: \"fdf4711c-418a-436b-848a-d46974566573\") "
Oct 30 19:16:58.116657 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.116500    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-model-cache\") pod \"60a58597-fdc4-4722-8d2a-29e475f84a76\" (UID: \"60a58597-fdc4-4722-8d2a-29e475f84a76\") "
Oct 30 19:16:58.116657 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.116569    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-tmp-dir\") pod \"fdf4711c-418a-436b-848a-d46974566573\" (UID: \"fdf4711c-418a-436b-848a-d46974566573\") "
Oct 30 19:16:58.116657 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.116634    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-dshm\" (UniqueName: \"kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-dshm\") pod \"60a58597-fdc4-4722-8d2a-29e475f84a76\" (UID: \"60a58597-fdc4-4722-8d2a-29e475f84a76\") "
Oct 30 19:16:58.116822 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.116694    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-model-cache\") pod \"fdf4711c-418a-436b-848a-d46974566573\" (UID: \"fdf4711c-418a-436b-848a-d46974566573\") "
Oct 30 19:16:58.116822 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.116725    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-tmp-dir\") pod \"60a58597-fdc4-4722-8d2a-29e475f84a76\" (UID: \"60a58597-fdc4-4722-8d2a-29e475f84a76\") "
Oct 30 19:16:58.116822 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.116804    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-home\" (UniqueName: \"kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-home\") pod \"fdf4711c-418a-436b-848a-d46974566573\" (UID: \"fdf4711c-418a-436b-848a-d46974566573\") "
Oct 30 19:16:58.116984 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.116855    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60a58597-fdc4-4722-8d2a-29e475f84a76-tls-certs\" (UniqueName: \"kubernetes.io/secret/60a58597-fdc4-4722-8d2a-29e475f84a76-tls-certs\") pod \"60a58597-fdc4-4722-8d2a-29e475f84a76\" (UID: \"60a58597-fdc4-4722-8d2a-29e475f84a76\") "
Oct 30 19:16:58.116984 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.116884    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-home\" (UniqueName: \"kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-home\") pod \"60a58597-fdc4-4722-8d2a-29e475f84a76\" (UID: \"60a58597-fdc4-4722-8d2a-29e475f84a76\") "
Oct 30 19:16:58.116984 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.116894    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-tmp-dir" pod "fdf4711c-418a-436b-848a-d46974566573" (UID: "fdf4711c-418a-436b-848a-d46974566573"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.117150 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.117021    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-model-cache" pod "fdf4711c-418a-436b-848a-d46974566573" (UID: "fdf4711c-418a-436b-848a-d46974566573"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.117150 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.117101    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-home" pod "fdf4711c-418a-436b-848a-d46974566573" (UID: "fdf4711c-418a-436b-848a-d46974566573"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.117311 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.117291    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-tmp-dir" pod "60a58597-fdc4-4722-8d2a-29e475f84a76" (UID: "60a58597-fdc4-4722-8d2a-29e475f84a76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.117399 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.117379    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-model-cache" pod "60a58597-fdc4-4722-8d2a-29e475f84a76" (UID: "60a58597-fdc4-4722-8d2a-29e475f84a76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.117610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.117521    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37d1db7a-108c-4972-8223-3ed0a626a6dc-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.117701 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.117608    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-home" pod "60a58597-fdc4-4722-8d2a-29e475f84a76" (UID: "60a58597-fdc4-4722-8d2a-29e475f84a76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.117701 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.117611    4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-model-cache\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.117701 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.117640    4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-tmp-dir\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.117701 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.117671    4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-home\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.117701 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.117689    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37d1db7a-108c-4972-8223-3ed0a626a6dc-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.117989 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.117708    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb3461be-6387-4cf8-8d71-082737fdd0d8-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.117989 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.117721    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmx29\" (UniqueName: \"kubernetes.io/projected/37d1db7a-108c-4972-8223-3ed0a626a6dc-kube-api-access-tmx29\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.117989 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.117735    4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-model-cache\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.117989 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.117749    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb3461be-6387-4cf8-8d71-082737fdd0d8-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.117989 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.117763    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjvsd\" (UniqueName: \"kubernetes.io/projected/fb3461be-6387-4cf8-8d71-082737fdd0d8-kube-api-access-qjvsd\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.117989 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.117777    4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-tmp-dir\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.119295 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.119257    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-dshm" pod "fdf4711c-418a-436b-848a-d46974566573" (UID: "fdf4711c-418a-436b-848a-d46974566573"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.119567 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.119538    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-dshm" pod "60a58597-fdc4-4722-8d2a-29e475f84a76" (UID: "60a58597-fdc4-4722-8d2a-29e475f84a76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.119694 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.119640    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60a58597-fdc4-4722-8d2a-29e475f84a76-tls-certs" pod "60a58597-fdc4-4722-8d2a-29e475f84a76" (UID: "60a58597-fdc4-4722-8d2a-29e475f84a76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:58.119934 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.119913    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf4711c-418a-436b-848a-d46974566573-tls-certs" pod "fdf4711c-418a-436b-848a-d46974566573" (UID: "fdf4711c-418a-436b-848a-d46974566573"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:58.120004 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.119990    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a58597-fdc4-4722-8d2a-29e475f84a76-kube-api-access-5s4bl" pod "60a58597-fdc4-4722-8d2a-29e475f84a76" (UID: "60a58597-fdc4-4722-8d2a-29e475f84a76"). InnerVolumeSpecName "kube-api-access-5s4bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:58.121015 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.120995    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf4711c-418a-436b-848a-d46974566573-kube-api-access-tvm88" pod "fdf4711c-418a-436b-848a-d46974566573" (UID: "fdf4711c-418a-436b-848a-d46974566573"). InnerVolumeSpecName "kube-api-access-tvm88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:58.218408 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.218326    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-kube-api-access-zxwmz\" (UniqueName: \"kubernetes.io/projected/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-kube-api-access-zxwmz\") pod \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\" (UID: \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\") "
Oct 30 19:16:58.218408 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.218372    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-home\" (UniqueName: \"kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-home\") pod \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\" (UID: \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\") "
Oct 30 19:16:58.218408 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.218394    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-tls-certs\" (UniqueName: \"kubernetes.io/secret/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-tls-certs\") pod \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\" (UID: \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\") "
Oct 30 19:16:58.218632 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.218465    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-dshm\") pod \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\" (UID: \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\") "
Oct 30 19:16:58.218632 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.218554    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-tmp-dir\") pod \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\" (UID: \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\") "
Oct 30 19:16:58.218632 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.218618    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-model-cache\") pod \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\" (UID: \"51f3e639-9cd4-4aaa-a568-f19b6d2704fd\") "
Oct 30 19:16:58.218784 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.218656    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-home" pod "51f3e639-9cd4-4aaa-a568-f19b6d2704fd" (UID: "51f3e639-9cd4-4aaa-a568-f19b6d2704fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.218892 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.218863    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-tmp-dir" pod "51f3e639-9cd4-4aaa-a568-f19b6d2704fd" (UID: "51f3e639-9cd4-4aaa-a568-f19b6d2704fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.219037 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.218928    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-model-cache" pod "51f3e639-9cd4-4aaa-a568-f19b6d2704fd" (UID: "51f3e639-9cd4-4aaa-a568-f19b6d2704fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.219196 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.219178    4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf4711c-418a-436b-848a-d46974566573-dshm\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.219268 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.219197    4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf4711c-418a-436b-848a-d46974566573-tls-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.219268 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.219206    4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-tmp-dir\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.219268 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.219215    4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-model-cache\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.219268 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.219227    4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-dshm\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.219268 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.219238    4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-home\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.219268 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.219248    4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60a58597-fdc4-4722-8d2a-29e475f84a76-tls-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.219268 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.219256    4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60a58597-fdc4-4722-8d2a-29e475f84a76-home\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.219268 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.219265    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s4bl\" (UniqueName: \"kubernetes.io/projected/60a58597-fdc4-4722-8d2a-29e475f84a76-kube-api-access-5s4bl\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.219724 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.219275    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvm88\" (UniqueName: \"kubernetes.io/projected/fdf4711c-418a-436b-848a-d46974566573-kube-api-access-tvm88\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.220643 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.220621    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-dshm" pod "51f3e639-9cd4-4aaa-a568-f19b6d2704fd" (UID: "51f3e639-9cd4-4aaa-a568-f19b6d2704fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:16:58.220760 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.220674    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-kube-api-access-zxwmz" pod "51f3e639-9cd4-4aaa-a568-f19b6d2704fd" (UID: "51f3e639-9cd4-4aaa-a568-f19b6d2704fd"). InnerVolumeSpecName "kube-api-access-zxwmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:16:58.220760 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.220691    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-tls-certs" pod "51f3e639-9cd4-4aaa-a568-f19b6d2704fd" (UID: "51f3e639-9cd4-4aaa-a568-f19b6d2704fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:16:58.320801 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.320765    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxwmz\" (UniqueName: \"kubernetes.io/projected/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-kube-api-access-zxwmz\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.320801 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.320795    4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-tls-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.320801 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.320806    4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51f3e639-9cd4-4aaa-a568-f19b6d2704fd-dshm\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:16:58.716287 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.716259    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da0a81c-92hnt"
Oct 30 19:16:58.716287 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.716277    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7da0a81c-92hnt" event={"ID":"37d1db7a-108c-4972-8223-3ed0a626a6dc","Type":"ContainerDied","Data":"a6757af70c65d29b1a2b3aec7918725e9ed0a057e75fb355b34a8239115cab98"}
Oct 30 19:16:58.716772 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.716327    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6757af70c65d29b1a2b3aec7918725e9ed0a057e75fb355b34a8239115cab98"
Oct 30 19:16:58.717770 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.717745    4924 generic.go:356] "Generic (PLEG): container finished" podID="51f3e639-9cd4-4aaa-a568-f19b6d2704fd" containerID="aa0f8e048bb1d418eac1c5eb0d71205e7830315a6f63f556312a1f9aa6c504fa" exitCode=0
Oct 30 19:16:58.717882 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.717783    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"
Oct 30 19:16:58.717882 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.717818    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj" event={"ID":"51f3e639-9cd4-4aaa-a568-f19b6d2704fd","Type":"ContainerDied","Data":"aa0f8e048bb1d418eac1c5eb0d71205e7830315a6f63f556312a1f9aa6c504fa"}
Oct 30 19:16:58.717882 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.717858    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj" event={"ID":"51f3e639-9cd4-4aaa-a568-f19b6d2704fd","Type":"ContainerDied","Data":"a30e5ad2631335ba7633794cfe6eb7f8800c53e439aea833eb459a4d81c578af"}
Oct 30 19:16:58.717882 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.717877    4924 scope.go:122] "RemoveContainer" containerID="aa0f8e048bb1d418eac1c5eb0d71205e7830315a6f63f556312a1f9aa6c504fa"
Oct 30 19:16:58.719365 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.719320    4924 generic.go:356] "Generic (PLEG): container finished" podID="60a58597-fdc4-4722-8d2a-29e475f84a76" containerID="480ca2e8a1a174a796513c83a2316f1e00d053c13352d0c7790b03d7aabd619a" exitCode=0
Oct 30 19:16:58.719448 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.719362    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"
Oct 30 19:16:58.719509 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.719398    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86" event={"ID":"60a58597-fdc4-4722-8d2a-29e475f84a76","Type":"ContainerDied","Data":"480ca2e8a1a174a796513c83a2316f1e00d053c13352d0c7790b03d7aabd619a"}
Oct 30 19:16:58.719564 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.719507    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86" event={"ID":"60a58597-fdc4-4722-8d2a-29e475f84a76","Type":"ContainerDied","Data":"bc886d1c332bd5311ee94af204769eb360f84a0dda4db7ca0214fc886c528c06"}
Oct 30 19:16:58.721104 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.721047    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeda332-g5dvx"
Oct 30 19:16:58.721104 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.721091    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeda332-g5dvx" event={"ID":"fb3461be-6387-4cf8-8d71-082737fdd0d8","Type":"ContainerDied","Data":"4a6eae6577bd1db2ded58d7a10dfb569b0c9baef2a37ea55cea0233762a93d10"}
Oct 30 19:16:58.721294 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.721138    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a6eae6577bd1db2ded58d7a10dfb569b0c9baef2a37ea55cea0233762a93d10"
Oct 30 19:16:58.722725 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.722604    4924 generic.go:356] "Generic (PLEG): container finished" podID="fdf4711c-418a-436b-848a-d46974566573" containerID="6a72e506dbbd782b6d51df437ebcbfd0e90ccd639153874a83f28c29827d33ad" exitCode=0
Oct 30 19:16:58.722725 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.722638    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2" event={"ID":"fdf4711c-418a-436b-848a-d46974566573","Type":"ContainerDied","Data":"6a72e506dbbd782b6d51df437ebcbfd0e90ccd639153874a83f28c29827d33ad"}
Oct 30 19:16:58.722725 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.722655    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"
Oct 30 19:16:58.722725 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.722663    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2" event={"ID":"fdf4711c-418a-436b-848a-d46974566573","Type":"ContainerDied","Data":"4a05f6deb91406b968c838237439280cb8ca45d514013b66962b88e3e15af471"}
Oct 30 19:16:58.733948 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.733933    4924 scope.go:122] "RemoveContainer" containerID="aa0f8e048bb1d418eac1c5eb0d71205e7830315a6f63f556312a1f9aa6c504fa"
Oct 30 19:16:58.734249 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:16:58.734230    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa0f8e048bb1d418eac1c5eb0d71205e7830315a6f63f556312a1f9aa6c504fa\": container with ID starting with aa0f8e048bb1d418eac1c5eb0d71205e7830315a6f63f556312a1f9aa6c504fa not found: ID does not exist" containerID="aa0f8e048bb1d418eac1c5eb0d71205e7830315a6f63f556312a1f9aa6c504fa"
Oct 30 19:16:58.734321 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.734255    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa0f8e048bb1d418eac1c5eb0d71205e7830315a6f63f556312a1f9aa6c504fa"} err="failed to get container status \"aa0f8e048bb1d418eac1c5eb0d71205e7830315a6f63f556312a1f9aa6c504fa\": rpc error: code = NotFound desc = could not find container \"aa0f8e048bb1d418eac1c5eb0d71205e7830315a6f63f556312a1f9aa6c504fa\": container with ID starting with aa0f8e048bb1d418eac1c5eb0d71205e7830315a6f63f556312a1f9aa6c504fa not found: ID does not exist"
Oct 30 19:16:58.734321 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.734271    4924 scope.go:122] "RemoveContainer" containerID="480ca2e8a1a174a796513c83a2316f1e00d053c13352d0c7790b03d7aabd619a"
Oct 30 19:16:58.743030 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.743015    4924 scope.go:122] "RemoveContainer" containerID="480ca2e8a1a174a796513c83a2316f1e00d053c13352d0c7790b03d7aabd619a"
Oct 30 19:16:58.743351 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:16:58.743334    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"480ca2e8a1a174a796513c83a2316f1e00d053c13352d0c7790b03d7aabd619a\": container with ID starting with 480ca2e8a1a174a796513c83a2316f1e00d053c13352d0c7790b03d7aabd619a not found: ID does not exist" containerID="480ca2e8a1a174a796513c83a2316f1e00d053c13352d0c7790b03d7aabd619a"
Oct 30 19:16:58.743404 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.743359    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"480ca2e8a1a174a796513c83a2316f1e00d053c13352d0c7790b03d7aabd619a"} err="failed to get container status \"480ca2e8a1a174a796513c83a2316f1e00d053c13352d0c7790b03d7aabd619a\": rpc error: code = NotFound desc = could not find container \"480ca2e8a1a174a796513c83a2316f1e00d053c13352d0c7790b03d7aabd619a\": container with ID starting with 480ca2e8a1a174a796513c83a2316f1e00d053c13352d0c7790b03d7aabd619a not found: ID does not exist"
Oct 30 19:16:58.743404 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.743374    4924 scope.go:122] "RemoveContainer" containerID="6a72e506dbbd782b6d51df437ebcbfd0e90ccd639153874a83f28c29827d33ad"
Oct 30 19:16:58.751913 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.751895    4924 scope.go:122] "RemoveContainer" containerID="6a72e506dbbd782b6d51df437ebcbfd0e90ccd639153874a83f28c29827d33ad"
Oct 30 19:16:58.752191 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:16:58.752172    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a72e506dbbd782b6d51df437ebcbfd0e90ccd639153874a83f28c29827d33ad\": container with ID starting with 6a72e506dbbd782b6d51df437ebcbfd0e90ccd639153874a83f28c29827d33ad not found: ID does not exist" containerID="6a72e506dbbd782b6d51df437ebcbfd0e90ccd639153874a83f28c29827d33ad"
Oct 30 19:16:58.752257 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.752198    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a72e506dbbd782b6d51df437ebcbfd0e90ccd639153874a83f28c29827d33ad"} err="failed to get container status \"6a72e506dbbd782b6d51df437ebcbfd0e90ccd639153874a83f28c29827d33ad\": rpc error: code = NotFound desc = could not find container \"6a72e506dbbd782b6d51df437ebcbfd0e90ccd639153874a83f28c29827d33ad\": container with ID starting with 6a72e506dbbd782b6d51df437ebcbfd0e90ccd639153874a83f28c29827d33ad not found: ID does not exist"
Oct 30 19:16:58.974920 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.974830    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"]
Oct 30 19:16:58.977983 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.977953    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86" podUID="60a58597-fdc4-4722-8d2a-29e475f84a76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 19:16:58.980009 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:58.979988    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ec8dab/distinct-w2-ec8dab-kserve-76788c545b-26h86"]
Oct 30 19:16:59.084030 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:59.083993    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"]
Oct 30 19:16:59.089279 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:59.089245    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ec8dab/simulator-w2-ec8dab-kserve-b87777d68-4l6n2"]
Oct 30 19:16:59.192815 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:59.192777    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"]
Oct 30 19:16:59.198222 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:59.198187    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ec8dab/distinct-2-w2-ec8dab-kserve-7767b4b86c-2j5mj"]
Oct 30 19:16:59.314996 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:16:59.314894    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:00.417198 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.417163    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf" podUID="dc3baffc-49c4-4101-9bf4-047229d89000" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:17:00.417598 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.417167    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d" podUID="b9cd1a58-ecd4-4611-aaac-28109cd2981a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:17:00.605796 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.605762    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15294a79-vdrpp"]
Oct 30 19:17:00.606311 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.606296    4924 cpu_manager.go:414] "containerMap: removing container" podUID="60a58597-fdc4-4722-8d2a-29e475f84a76" containerName="main"
Oct 30 19:17:00.606376 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.606312    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60a58597-fdc4-4722-8d2a-29e475f84a76" containerName="main"
Oct 30 19:17:00.606376 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.606337    4924 cpu_manager.go:414] "containerMap: removing container" podUID="fb3461be-6387-4cf8-8d71-082737fdd0d8" containerName="revoke-keys"
Oct 30 19:17:00.606376 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.606343    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb3461be-6387-4cf8-8d71-082737fdd0d8" containerName="revoke-keys"
Oct 30 19:17:00.606376 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.606356    4924 cpu_manager.go:414] "containerMap: removing container" podUID="37d1db7a-108c-4972-8223-3ed0a626a6dc" containerName="revoke-keys"
Oct 30 19:17:00.606376 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.606361    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37d1db7a-108c-4972-8223-3ed0a626a6dc" containerName="revoke-keys"
Oct 30 19:17:00.606376 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.606376    4924 cpu_manager.go:414] "containerMap: removing container" podUID="51f3e639-9cd4-4aaa-a568-f19b6d2704fd" containerName="main"
Oct 30 19:17:00.606565 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.606381    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51f3e639-9cd4-4aaa-a568-f19b6d2704fd" containerName="main"
Oct 30 19:17:00.606565 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.606393    4924 cpu_manager.go:414] "containerMap: removing container" podUID="fdf4711c-418a-436b-848a-d46974566573" containerName="main"
Oct 30 19:17:00.606565 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.606397    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdf4711c-418a-436b-848a-d46974566573" containerName="main"
Oct 30 19:17:00.606565 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.606470    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="51f3e639-9cd4-4aaa-a568-f19b6d2704fd" containerName="main"
Oct 30 19:17:00.606565 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.606479    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdf4711c-418a-436b-848a-d46974566573" containerName="main"
Oct 30 19:17:00.606565 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.606487    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="60a58597-fdc4-4722-8d2a-29e475f84a76" containerName="main"
Oct 30 19:17:00.606565 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.606496    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb3461be-6387-4cf8-8d71-082737fdd0d8" containerName="revoke-keys"
Oct 30 19:17:00.606565 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.606502    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="37d1db7a-108c-4972-8223-3ed0a626a6dc" containerName="revoke-keys"
Oct 30 19:17:00.611463 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.611442    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15294a79-vdrpp"
Oct 30 19:17:00.655581 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.655551    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37d4b500-hpn5f"]
Oct 30 19:17:00.659514 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.659494    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d4b500-hpn5f"
Oct 30 19:17:00.699013 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.698941    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:17:00.702230 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.702207    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15294a79-vdrpp"]
Oct 30 19:17:00.711544 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.711521    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:17:00.711741 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.711723    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:17:00.712076 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.712027    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:17:00.715856 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.715831    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37d4b500-hpn5f"]
Oct 30 19:17:00.754304 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.754276    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wntpk\" (UniqueName: \"kubernetes.io/projected/96230ce5-7a7e-48f9-a87f-1b786d7f8730-kube-api-access-wntpk\") pod \"maas-api-revoke-sub-15294a79-vdrpp\" (UID: \"96230ce5-7a7e-48f9-a87f-1b786d7f8730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15294a79-vdrpp"
Oct 30 19:17:00.754432 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.754395    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96230ce5-7a7e-48f9-a87f-1b786d7f8730-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15294a79-vdrpp\" (UID: \"96230ce5-7a7e-48f9-a87f-1b786d7f8730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15294a79-vdrpp"
Oct 30 19:17:00.754432 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.754422    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04617682-2d5b-421c-82c4-e7e60c01edbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37d4b500-hpn5f\" (UID: \"04617682-2d5b-421c-82c4-e7e60c01edbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d4b500-hpn5f"
Oct 30 19:17:00.754570 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.754549    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5s5h\" (UniqueName: \"kubernetes.io/projected/04617682-2d5b-421c-82c4-e7e60c01edbf-kube-api-access-k5s5h\") pod \"maas-api-revoke-sub-37d4b500-hpn5f\" (UID: \"04617682-2d5b-421c-82c4-e7e60c01edbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d4b500-hpn5f"
Oct 30 19:17:00.754693 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.754680    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96230ce5-7a7e-48f9-a87f-1b786d7f8730-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15294a79-vdrpp\" (UID: \"96230ce5-7a7e-48f9-a87f-1b786d7f8730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15294a79-vdrpp"
Oct 30 19:17:00.754732 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.754703    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04617682-2d5b-421c-82c4-e7e60c01edbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37d4b500-hpn5f\" (UID: \"04617682-2d5b-421c-82c4-e7e60c01edbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d4b500-hpn5f"
Oct 30 19:17:00.835784 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.835752    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51f3e639-9cd4-4aaa-a568-f19b6d2704fd" path="/var/lib/kubelet/pods/51f3e639-9cd4-4aaa-a568-f19b6d2704fd/volumes"
Oct 30 19:17:00.836251 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.836236    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60a58597-fdc4-4722-8d2a-29e475f84a76" path="/var/lib/kubelet/pods/60a58597-fdc4-4722-8d2a-29e475f84a76/volumes"
Oct 30 19:17:00.836655 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.836641    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdf4711c-418a-436b-848a-d46974566573" path="/var/lib/kubelet/pods/fdf4711c-418a-436b-848a-d46974566573/volumes"
Oct 30 19:17:00.855366 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.855333    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96230ce5-7a7e-48f9-a87f-1b786d7f8730-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15294a79-vdrpp\" (UID: \"96230ce5-7a7e-48f9-a87f-1b786d7f8730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15294a79-vdrpp"
Oct 30 19:17:00.855534 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.855375    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04617682-2d5b-421c-82c4-e7e60c01edbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37d4b500-hpn5f\" (UID: \"04617682-2d5b-421c-82c4-e7e60c01edbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d4b500-hpn5f"
Oct 30 19:17:00.855534 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.855409    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5s5h\" (UniqueName: \"kubernetes.io/projected/04617682-2d5b-421c-82c4-e7e60c01edbf-kube-api-access-k5s5h\") pod \"maas-api-revoke-sub-37d4b500-hpn5f\" (UID: \"04617682-2d5b-421c-82c4-e7e60c01edbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d4b500-hpn5f"
Oct 30 19:17:00.855534 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.855501    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96230ce5-7a7e-48f9-a87f-1b786d7f8730-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15294a79-vdrpp\" (UID: \"96230ce5-7a7e-48f9-a87f-1b786d7f8730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15294a79-vdrpp"
Oct 30 19:17:00.855534 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.855522    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04617682-2d5b-421c-82c4-e7e60c01edbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37d4b500-hpn5f\" (UID: \"04617682-2d5b-421c-82c4-e7e60c01edbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d4b500-hpn5f"
Oct 30 19:17:00.855816 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.855563    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wntpk\" (UniqueName: \"kubernetes.io/projected/96230ce5-7a7e-48f9-a87f-1b786d7f8730-kube-api-access-wntpk\") pod \"maas-api-revoke-sub-15294a79-vdrpp\" (UID: \"96230ce5-7a7e-48f9-a87f-1b786d7f8730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15294a79-vdrpp"
Oct 30 19:17:00.856212 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.856187    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04617682-2d5b-421c-82c4-e7e60c01edbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37d4b500-hpn5f\" (UID: \"04617682-2d5b-421c-82c4-e7e60c01edbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d4b500-hpn5f"
Oct 30 19:17:00.856697 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.856667    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96230ce5-7a7e-48f9-a87f-1b786d7f8730-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15294a79-vdrpp\" (UID: \"96230ce5-7a7e-48f9-a87f-1b786d7f8730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15294a79-vdrpp"
Oct 30 19:17:00.858639 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.858616    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04617682-2d5b-421c-82c4-e7e60c01edbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37d4b500-hpn5f\" (UID: \"04617682-2d5b-421c-82c4-e7e60c01edbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d4b500-hpn5f"
Oct 30 19:17:00.858736 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.858620    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96230ce5-7a7e-48f9-a87f-1b786d7f8730-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15294a79-vdrpp\" (UID: \"96230ce5-7a7e-48f9-a87f-1b786d7f8730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15294a79-vdrpp"
Oct 30 19:17:00.911337 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.911304    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wntpk\" (UniqueName: \"kubernetes.io/projected/96230ce5-7a7e-48f9-a87f-1b786d7f8730-kube-api-access-wntpk\") pod \"maas-api-revoke-sub-15294a79-vdrpp\" (UID: \"96230ce5-7a7e-48f9-a87f-1b786d7f8730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15294a79-vdrpp"
Oct 30 19:17:00.911589 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.911567    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5s5h\" (UniqueName: \"kubernetes.io/projected/04617682-2d5b-421c-82c4-e7e60c01edbf-kube-api-access-k5s5h\") pod \"maas-api-revoke-sub-37d4b500-hpn5f\" (UID: \"04617682-2d5b-421c-82c4-e7e60c01edbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d4b500-hpn5f"
Oct 30 19:17:00.925380 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.925357    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15294a79-vdrpp"
Oct 30 19:17:00.974164 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.974083    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d4b500-hpn5f"
Oct 30 19:17:00.997693 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:00.997664    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:01.298691 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:01.298662    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15294a79-vdrpp"]
Oct 30 19:17:01.299529 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:17:01.299500    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96230ce5_7a7e_48f9_a87f_1b786d7f8730.slice/crio-2332718df82816362f44420c3a2e494f2d8a2731edd0b17d41f1e77e3dc7b06a WatchSource:0}: Error finding container 2332718df82816362f44420c3a2e494f2d8a2731edd0b17d41f1e77e3dc7b06a: Status 404 returned error can't find the container with id 2332718df82816362f44420c3a2e494f2d8a2731edd0b17d41f1e77e3dc7b06a
Oct 30 19:17:01.356651 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:01.356623    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37d4b500-hpn5f"]
Oct 30 19:17:01.357155 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:17:01.357120    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04617682_2d5b_421c_82c4_e7e60c01edbf.slice/crio-1702a7959d5c61cd2e62131ca93b9448f5144f35df4722a3acb1d63375272f3f WatchSource:0}: Error finding container 1702a7959d5c61cd2e62131ca93b9448f5144f35df4722a3acb1d63375272f3f: Status 404 returned error can't find the container with id 1702a7959d5c61cd2e62131ca93b9448f5144f35df4722a3acb1d63375272f3f
Oct 30 19:17:01.396685 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:01.396656    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:01.625469 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:01.625419    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n"]
Oct 30 19:17:01.628474 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:01.628451    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a1617e-ks45n"]
Oct 30 19:17:01.749645 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:01.749604    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d4b500-hpn5f" event={"ID":"04617682-2d5b-421c-82c4-e7e60c01edbf","Type":"ContainerStarted","Data":"f09b3b79c9f6fe950aa23a51e89ecc97e9a0cf1b0d5deeb9c6be3187070ff89b"}
Oct 30 19:17:01.749899 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:01.749668    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d4b500-hpn5f" event={"ID":"04617682-2d5b-421c-82c4-e7e60c01edbf","Type":"ContainerStarted","Data":"1702a7959d5c61cd2e62131ca93b9448f5144f35df4722a3acb1d63375272f3f"}
Oct 30 19:17:01.751440 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:01.751415    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15294a79-vdrpp" event={"ID":"96230ce5-7a7e-48f9-a87f-1b786d7f8730","Type":"ContainerStarted","Data":"d01c0ff788f2902f6fc2b616f53ee131f6a9fd426a4c3c7127539ebb25e31611"}
Oct 30 19:17:01.751579 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:01.751459    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15294a79-vdrpp" event={"ID":"96230ce5-7a7e-48f9-a87f-1b786d7f8730","Type":"ContainerStarted","Data":"2332718df82816362f44420c3a2e494f2d8a2731edd0b17d41f1e77e3dc7b06a"}
Oct 30 19:17:01.847554 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:01.847520    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:01.863419 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:01.863257    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d4b500-hpn5f" podStartSLOduration=1.8632405410000001 podStartE2EDuration="1.863240541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:01.862149407 +0000 UTC m=+2423.139199852" watchObservedRunningTime="2026-10-30 19:17:01.863240541 +0000 UTC m=+2423.140290987"
Oct 30 19:17:01.973412 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:01.973362    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15294a79-vdrpp" podStartSLOduration=1.973349111 podStartE2EDuration="1.973349111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:01.971186386 +0000 UTC m=+2423.248236831" watchObservedRunningTime="2026-10-30 19:17:01.973349111 +0000 UTC m=+2423.250399555"
Oct 30 19:17:02.120939 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.120883    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c"
Oct 30 19:17:02.190809 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.190719    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:02.274651 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.274611    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0580aebc-4e9e-4578-872c-943719ee559f-kube-api-access-gmtcd\" (UniqueName: \"kubernetes.io/projected/0580aebc-4e9e-4578-872c-943719ee559f-kube-api-access-gmtcd\") pod \"0580aebc-4e9e-4578-872c-943719ee559f\" (UID: \"0580aebc-4e9e-4578-872c-943719ee559f\") "
Oct 30 19:17:02.274826 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.274802    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0580aebc-4e9e-4578-872c-943719ee559f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0580aebc-4e9e-4578-872c-943719ee559f-plugins-config-volume\") pod \"0580aebc-4e9e-4578-872c-943719ee559f\" (UID: \"0580aebc-4e9e-4578-872c-943719ee559f\") "
Oct 30 19:17:02.275230 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.275204    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0580aebc-4e9e-4578-872c-943719ee559f-plugins-config-volume" pod "0580aebc-4e9e-4578-872c-943719ee559f" (UID: "0580aebc-4e9e-4578-872c-943719ee559f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:02.277058 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.277029    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0580aebc-4e9e-4578-872c-943719ee559f-kube-api-access-gmtcd" pod "0580aebc-4e9e-4578-872c-943719ee559f" (UID: "0580aebc-4e9e-4578-872c-943719ee559f"). InnerVolumeSpecName "kube-api-access-gmtcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:02.281627 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.281602    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4"
Oct 30 19:17:02.375633 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.375603    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3dd0455-39e6-4b1b-80ee-ffbde02d4f00-kube-api-access-xd8qx\" (UniqueName: \"kubernetes.io/projected/d3dd0455-39e6-4b1b-80ee-ffbde02d4f00-kube-api-access-xd8qx\") pod \"d3dd0455-39e6-4b1b-80ee-ffbde02d4f00\" (UID: \"d3dd0455-39e6-4b1b-80ee-ffbde02d4f00\") "
Oct 30 19:17:02.375833 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.375809    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3dd0455-39e6-4b1b-80ee-ffbde02d4f00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3dd0455-39e6-4b1b-80ee-ffbde02d4f00-plugins-config-volume\") pod \"d3dd0455-39e6-4b1b-80ee-ffbde02d4f00\" (UID: \"d3dd0455-39e6-4b1b-80ee-ffbde02d4f00\") "
Oct 30 19:17:02.376242 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.376218    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmtcd\" (UniqueName: \"kubernetes.io/projected/0580aebc-4e9e-4578-872c-943719ee559f-kube-api-access-gmtcd\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:02.376315 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.376246    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0580aebc-4e9e-4578-872c-943719ee559f-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:02.376315 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.376246    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3dd0455-39e6-4b1b-80ee-ffbde02d4f00-plugins-config-volume" pod "d3dd0455-39e6-4b1b-80ee-ffbde02d4f00" (UID: "d3dd0455-39e6-4b1b-80ee-ffbde02d4f00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:02.377674 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.377650    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3dd0455-39e6-4b1b-80ee-ffbde02d4f00-kube-api-access-xd8qx" pod "d3dd0455-39e6-4b1b-80ee-ffbde02d4f00" (UID: "d3dd0455-39e6-4b1b-80ee-ffbde02d4f00"). InnerVolumeSpecName "kube-api-access-xd8qx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:02.477597 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.477505    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd8qx\" (UniqueName: \"kubernetes.io/projected/d3dd0455-39e6-4b1b-80ee-ffbde02d4f00-kube-api-access-xd8qx\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:02.477597 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.477542    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3dd0455-39e6-4b1b-80ee-ffbde02d4f00-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:02.760902 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.760810    4924 generic.go:356] "Generic (PLEG): container finished" podID="0580aebc-4e9e-4578-872c-943719ee559f" containerID="5e4b153de2dc19cbf358ec5addca655afdce7fe5f1471a8003e66065b16c1282" exitCode=0
Oct 30 19:17:02.760902 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.760856    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c"
Oct 30 19:17:02.760902 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.760886    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c" event={"ID":"0580aebc-4e9e-4578-872c-943719ee559f","Type":"ContainerDied","Data":"5e4b153de2dc19cbf358ec5addca655afdce7fe5f1471a8003e66065b16c1282"}
Oct 30 19:17:02.761497 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.760965    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c" event={"ID":"0580aebc-4e9e-4578-872c-943719ee559f","Type":"ContainerDied","Data":"c53bf04532a0ff7b0ca80117dcb68f5e5baaf2b6354ed4659417ad2ff150ef95"}
Oct 30 19:17:02.761497 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.760989    4924 scope.go:122] "RemoveContainer" containerID="5e4b153de2dc19cbf358ec5addca655afdce7fe5f1471a8003e66065b16c1282"
Oct 30 19:17:02.762443 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.762420    4924 generic.go:356] "Generic (PLEG): container finished" podID="d3dd0455-39e6-4b1b-80ee-ffbde02d4f00" containerID="389d884c1ef4feb0839e774bb9640ea12d9d699110005bcda5dda8022a41cf29" exitCode=0
Oct 30 19:17:02.762578 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.762460    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4"
Oct 30 19:17:02.762578 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.762558    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4" event={"ID":"d3dd0455-39e6-4b1b-80ee-ffbde02d4f00","Type":"ContainerDied","Data":"389d884c1ef4feb0839e774bb9640ea12d9d699110005bcda5dda8022a41cf29"}
Oct 30 19:17:02.762702 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.762615    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4" event={"ID":"d3dd0455-39e6-4b1b-80ee-ffbde02d4f00","Type":"ContainerDied","Data":"aa47952eeab34e5e76b599b894c1c6d7bf08a5480f9396a89c594183b9cf9d47"}
Oct 30 19:17:02.764506 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.764485    4924 generic.go:356] "Generic (PLEG): container finished" podID="96230ce5-7a7e-48f9-a87f-1b786d7f8730" containerID="d01c0ff788f2902f6fc2b616f53ee131f6a9fd426a4c3c7127539ebb25e31611" exitCode=0
Oct 30 19:17:02.764636 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.764569    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15294a79-vdrpp" event={"ID":"96230ce5-7a7e-48f9-a87f-1b786d7f8730","Type":"ContainerDied","Data":"d01c0ff788f2902f6fc2b616f53ee131f6a9fd426a4c3c7127539ebb25e31611"}
Oct 30 19:17:02.766516 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.766491    4924 generic.go:356] "Generic (PLEG): container finished" podID="04617682-2d5b-421c-82c4-e7e60c01edbf" containerID="f09b3b79c9f6fe950aa23a51e89ecc97e9a0cf1b0d5deeb9c6be3187070ff89b" exitCode=0
Oct 30 19:17:02.766620 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.766572    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d4b500-hpn5f" event={"ID":"04617682-2d5b-421c-82c4-e7e60c01edbf","Type":"ContainerDied","Data":"f09b3b79c9f6fe950aa23a51e89ecc97e9a0cf1b0d5deeb9c6be3187070ff89b"}
Oct 30 19:17:02.773982 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.773959    4924 scope.go:122] "RemoveContainer" containerID="5e4b153de2dc19cbf358ec5addca655afdce7fe5f1471a8003e66065b16c1282"
Oct 30 19:17:02.774341 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:17:02.774319    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e4b153de2dc19cbf358ec5addca655afdce7fe5f1471a8003e66065b16c1282\": container with ID starting with 5e4b153de2dc19cbf358ec5addca655afdce7fe5f1471a8003e66065b16c1282 not found: ID does not exist" containerID="5e4b153de2dc19cbf358ec5addca655afdce7fe5f1471a8003e66065b16c1282"
Oct 30 19:17:02.774439 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.774348    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e4b153de2dc19cbf358ec5addca655afdce7fe5f1471a8003e66065b16c1282"} err="failed to get container status \"5e4b153de2dc19cbf358ec5addca655afdce7fe5f1471a8003e66065b16c1282\": rpc error: code = NotFound desc = could not find container \"5e4b153de2dc19cbf358ec5addca655afdce7fe5f1471a8003e66065b16c1282\": container with ID starting with 5e4b153de2dc19cbf358ec5addca655afdce7fe5f1471a8003e66065b16c1282 not found: ID does not exist"
Oct 30 19:17:02.774439 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.774366    4924 scope.go:122] "RemoveContainer" containerID="389d884c1ef4feb0839e774bb9640ea12d9d699110005bcda5dda8022a41cf29"
Oct 30 19:17:02.784013 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.783988    4924 scope.go:122] "RemoveContainer" containerID="389d884c1ef4feb0839e774bb9640ea12d9d699110005bcda5dda8022a41cf29"
Oct 30 19:17:02.784324 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:17:02.784301    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"389d884c1ef4feb0839e774bb9640ea12d9d699110005bcda5dda8022a41cf29\": container with ID starting with 389d884c1ef4feb0839e774bb9640ea12d9d699110005bcda5dda8022a41cf29 not found: ID does not exist" containerID="389d884c1ef4feb0839e774bb9640ea12d9d699110005bcda5dda8022a41cf29"
Oct 30 19:17:02.784414 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.784335    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"389d884c1ef4feb0839e774bb9640ea12d9d699110005bcda5dda8022a41cf29"} err="failed to get container status \"389d884c1ef4feb0839e774bb9640ea12d9d699110005bcda5dda8022a41cf29\": rpc error: code = NotFound desc = could not find container \"389d884c1ef4feb0839e774bb9640ea12d9d699110005bcda5dda8022a41cf29\": container with ID starting with 389d884c1ef4feb0839e774bb9640ea12d9d699110005bcda5dda8022a41cf29 not found: ID does not exist"
Oct 30 19:17:02.835079 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:02.835035    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cab88e32-0265-482d-b3f1-a2e33f1dd3ac" path="/var/lib/kubelet/pods/cab88e32-0265-482d-b3f1-a2e33f1dd3ac/volumes"
Oct 30 19:17:03.166085 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:03.145260    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c"]
Oct 30 19:17:03.191966 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:03.191923    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-75e91b-78b44996cb-fwm9c"]
Oct 30 19:17:03.257840 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:03.257807    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4"]
Oct 30 19:17:03.263788 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:03.263757    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-75e91b-78c6c49dc-2lrd4"]
Oct 30 19:17:03.375376 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:03.375341    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:03.719509 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:03.719451    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:04.015935 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.015911    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15294a79-vdrpp"
Oct 30 19:17:04.018046 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.018028    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d4b500-hpn5f"
Oct 30 19:17:04.100080 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.100028    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96230ce5-7a7e-48f9-a87f-1b786d7f8730-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96230ce5-7a7e-48f9-a87f-1b786d7f8730-maas-api-openshift-ca\") pod \"96230ce5-7a7e-48f9-a87f-1b786d7f8730\" (UID: \"96230ce5-7a7e-48f9-a87f-1b786d7f8730\") "
Oct 30 19:17:04.100292 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.100120    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96230ce5-7a7e-48f9-a87f-1b786d7f8730-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96230ce5-7a7e-48f9-a87f-1b786d7f8730-maas-api-xks-ca\") pod \"96230ce5-7a7e-48f9-a87f-1b786d7f8730\" (UID: \"96230ce5-7a7e-48f9-a87f-1b786d7f8730\") "
Oct 30 19:17:04.100292 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.100230    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96230ce5-7a7e-48f9-a87f-1b786d7f8730-kube-api-access-wntpk\" (UniqueName: \"kubernetes.io/projected/96230ce5-7a7e-48f9-a87f-1b786d7f8730-kube-api-access-wntpk\") pod \"96230ce5-7a7e-48f9-a87f-1b786d7f8730\" (UID: \"96230ce5-7a7e-48f9-a87f-1b786d7f8730\") "
Oct 30 19:17:04.100429 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.100399    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96230ce5-7a7e-48f9-a87f-1b786d7f8730-maas-api-openshift-ca" pod "96230ce5-7a7e-48f9-a87f-1b786d7f8730" (UID: "96230ce5-7a7e-48f9-a87f-1b786d7f8730"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:04.100728 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.100699    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96230ce5-7a7e-48f9-a87f-1b786d7f8730-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:04.102285 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.102262    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96230ce5-7a7e-48f9-a87f-1b786d7f8730-maas-api-xks-ca" pod "96230ce5-7a7e-48f9-a87f-1b786d7f8730" (UID: "96230ce5-7a7e-48f9-a87f-1b786d7f8730"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:04.102285 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.102259    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96230ce5-7a7e-48f9-a87f-1b786d7f8730-kube-api-access-wntpk" pod "96230ce5-7a7e-48f9-a87f-1b786d7f8730" (UID: "96230ce5-7a7e-48f9-a87f-1b786d7f8730"). InnerVolumeSpecName "kube-api-access-wntpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:04.120960 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.120927    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:04.201414 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.201379    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04617682-2d5b-421c-82c4-e7e60c01edbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04617682-2d5b-421c-82c4-e7e60c01edbf-maas-api-xks-ca\") pod \"04617682-2d5b-421c-82c4-e7e60c01edbf\" (UID: \"04617682-2d5b-421c-82c4-e7e60c01edbf\") "
Oct 30 19:17:04.201593 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.201492    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04617682-2d5b-421c-82c4-e7e60c01edbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04617682-2d5b-421c-82c4-e7e60c01edbf-maas-api-openshift-ca\") pod \"04617682-2d5b-421c-82c4-e7e60c01edbf\" (UID: \"04617682-2d5b-421c-82c4-e7e60c01edbf\") "
Oct 30 19:17:04.201593 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.201520    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04617682-2d5b-421c-82c4-e7e60c01edbf-kube-api-access-k5s5h\" (UniqueName: \"kubernetes.io/projected/04617682-2d5b-421c-82c4-e7e60c01edbf-kube-api-access-k5s5h\") pod \"04617682-2d5b-421c-82c4-e7e60c01edbf\" (UID: \"04617682-2d5b-421c-82c4-e7e60c01edbf\") "
Oct 30 19:17:04.201904 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.201878    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04617682-2d5b-421c-82c4-e7e60c01edbf-maas-api-openshift-ca" pod "04617682-2d5b-421c-82c4-e7e60c01edbf" (UID: "04617682-2d5b-421c-82c4-e7e60c01edbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:04.202048 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.202032    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04617682-2d5b-421c-82c4-e7e60c01edbf-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:04.202186 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.202054    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wntpk\" (UniqueName: \"kubernetes.io/projected/96230ce5-7a7e-48f9-a87f-1b786d7f8730-kube-api-access-wntpk\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:04.202186 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.202086    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96230ce5-7a7e-48f9-a87f-1b786d7f8730-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:04.203634 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.203613    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04617682-2d5b-421c-82c4-e7e60c01edbf-maas-api-xks-ca" pod "04617682-2d5b-421c-82c4-e7e60c01edbf" (UID: "04617682-2d5b-421c-82c4-e7e60c01edbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:04.203634 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.203624    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04617682-2d5b-421c-82c4-e7e60c01edbf-kube-api-access-k5s5h" pod "04617682-2d5b-421c-82c4-e7e60c01edbf" (UID: "04617682-2d5b-421c-82c4-e7e60c01edbf"). InnerVolumeSpecName "kube-api-access-k5s5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:04.303439 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.303350    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04617682-2d5b-421c-82c4-e7e60c01edbf-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:04.303439 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.303391    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5s5h\" (UniqueName: \"kubernetes.io/projected/04617682-2d5b-421c-82c4-e7e60c01edbf-kube-api-access-k5s5h\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:04.325406 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.325372    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"]
Oct 30 19:17:04.325828 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.325584    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw" podUID="555228e6-5a19-4924-a121-9e4d321c0104" containerName="istio-proxy" containerID="cri-o://fc019e14612d4637ca7db4f3e772ec5ee0981722a41da1979884e155d5c85e6e" gracePeriod=30
Oct 30 19:17:04.353515 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.353486    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"]
Oct 30 19:17:04.353725 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.353705    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9" podUID="371f8b40-cf29-431e-be68-d3c7ce0eab02" containerName="istio-proxy" containerID="cri-o://1818fe2ebe802940fb40d3282c59ab94189fd7cd225e0666a90047646f79485c" gracePeriod=30
Oct 30 19:17:04.359380 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.359354    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"]
Oct 30 19:17:04.359577 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.359554    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4" podUID="34ce2d7c-43f1-41cc-9fb4-2f2e058a4609" containerName="istio-proxy" containerID="cri-o://a0507c3c12405c1dc1082bc99c2fc7b345b63590d6056dbaff0873cb5512c70a" gracePeriod=30
Oct 30 19:17:04.522373 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.522339    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:04.784612 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.784580    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15294a79-vdrpp"
Oct 30 19:17:04.784612 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.784602    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15294a79-vdrpp" event={"ID":"96230ce5-7a7e-48f9-a87f-1b786d7f8730","Type":"ContainerDied","Data":"2332718df82816362f44420c3a2e494f2d8a2731edd0b17d41f1e77e3dc7b06a"}
Oct 30 19:17:04.784848 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.784651    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2332718df82816362f44420c3a2e494f2d8a2731edd0b17d41f1e77e3dc7b06a"
Oct 30 19:17:04.785981 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.785963    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d4b500-hpn5f"
Oct 30 19:17:04.786123 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.785957    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37d4b500-hpn5f" event={"ID":"04617682-2d5b-421c-82c4-e7e60c01edbf","Type":"ContainerDied","Data":"1702a7959d5c61cd2e62131ca93b9448f5144f35df4722a3acb1d63375272f3f"}
Oct 30 19:17:04.786123 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.786098    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1702a7959d5c61cd2e62131ca93b9448f5144f35df4722a3acb1d63375272f3f"
Oct 30 19:17:04.841185 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.841145    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0580aebc-4e9e-4578-872c-943719ee559f" path="/var/lib/kubelet/pods/0580aebc-4e9e-4578-872c-943719ee559f/volumes"
Oct 30 19:17:04.841636 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:04.841616    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3dd0455-39e6-4b1b-80ee-ffbde02d4f00" path="/var/lib/kubelet/pods/d3dd0455-39e6-4b1b-80ee-ffbde02d4f00/volumes"
Oct 30 19:17:05.145912 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:05.145878    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:06.900057 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:06.900026    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:07.665193 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:07.665144    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:08.080444 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:08.080355    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:08.987875 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:08.987835    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:09.321129 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.321007    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:09.369755 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.369719    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"]
Oct 30 19:17:09.369998 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.369978    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w" podUID="d6ec64bd-1810-4f78-b451-9faae648be6e" containerName="istio-proxy" containerID="cri-o://46ba7f99d80df3f1099875ba25165476edd2516c73f2a949109eb012a3423062" gracePeriod=30
Oct 30 19:17:09.670624 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.670582    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:09.675481 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.675454    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:17:09.678256 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.678226    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-workload-socket\") pod \"371f8b40-cf29-431e-be68-d3c7ce0eab02\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") "
Oct 30 19:17:09.678396 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.678375    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/371f8b40-cf29-431e-be68-d3c7ce0eab02-kube-api-access-ghhkq\" (UniqueName: \"kubernetes.io/projected/371f8b40-cf29-431e-be68-d3c7ce0eab02-kube-api-access-ghhkq\") pod \"371f8b40-cf29-431e-be68-d3c7ce0eab02\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") "
Oct 30 19:17:09.678477 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.678432    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-token\" (UniqueName: \"kubernetes.io/projected/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-token\") pod \"371f8b40-cf29-431e-be68-d3c7ce0eab02\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") "
Oct 30 19:17:09.678477 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.678471    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-podinfo\") pod \"371f8b40-cf29-431e-be68-d3c7ce0eab02\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") "
Oct 30 19:17:09.678681 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.678519    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-workload-socket" pod "371f8b40-cf29-431e-be68-d3c7ce0eab02" (UID: "371f8b40-cf29-431e-be68-d3c7ce0eab02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:09.678681 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.678530    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-credential-socket\") pod \"371f8b40-cf29-431e-be68-d3c7ce0eab02\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") "
Oct 30 19:17:09.678681 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.678620    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-workload-certs\") pod \"371f8b40-cf29-431e-be68-d3c7ce0eab02\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") "
Oct 30 19:17:09.678825 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.678750    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-data\") pod \"371f8b40-cf29-431e-be68-d3c7ce0eab02\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") "
Oct 30 19:17:09.678825 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.678787    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-credential-socket" pod "371f8b40-cf29-431e-be68-d3c7ce0eab02" (UID: "371f8b40-cf29-431e-be68-d3c7ce0eab02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:09.678825 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.678801    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/371f8b40-cf29-431e-be68-d3c7ce0eab02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371f8b40-cf29-431e-be68-d3c7ce0eab02-istiod-ca-cert\") pod \"371f8b40-cf29-431e-be68-d3c7ce0eab02\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") "
Oct 30 19:17:09.678942 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.678836    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-envoy\") pod \"371f8b40-cf29-431e-be68-d3c7ce0eab02\" (UID: \"371f8b40-cf29-431e-be68-d3c7ce0eab02\") "
Oct 30 19:17:09.679401 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.679376    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.679401 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.679392    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-data" pod "371f8b40-cf29-431e-be68-d3c7ce0eab02" (UID: "371f8b40-cf29-431e-be68-d3c7ce0eab02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:09.679566 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.679404    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.679712 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.679685    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/371f8b40-cf29-431e-be68-d3c7ce0eab02-istiod-ca-cert" pod "371f8b40-cf29-431e-be68-d3c7ce0eab02" (UID: "371f8b40-cf29-431e-be68-d3c7ce0eab02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:09.680162 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.680136    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-workload-certs" pod "371f8b40-cf29-431e-be68-d3c7ce0eab02" (UID: "371f8b40-cf29-431e-be68-d3c7ce0eab02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:09.681696 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.681661    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-token" pod "371f8b40-cf29-431e-be68-d3c7ce0eab02" (UID: "371f8b40-cf29-431e-be68-d3c7ce0eab02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:09.682245 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.682220    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-envoy" pod "371f8b40-cf29-431e-be68-d3c7ce0eab02" (UID: "371f8b40-cf29-431e-be68-d3c7ce0eab02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:09.686279 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.686252    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-podinfo" pod "371f8b40-cf29-431e-be68-d3c7ce0eab02" (UID: "371f8b40-cf29-431e-be68-d3c7ce0eab02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:09.686524 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.686409    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371f8b40-cf29-431e-be68-d3c7ce0eab02-kube-api-access-ghhkq" pod "371f8b40-cf29-431e-be68-d3c7ce0eab02" (UID: "371f8b40-cf29-431e-be68-d3c7ce0eab02"). InnerVolumeSpecName "kube-api-access-ghhkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:09.702903 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.702883    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:17:09.723689 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.723669    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:17:09.780161 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.780131    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-credential-socket\") pod \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") "
Oct 30 19:17:09.780330 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.780175    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-istio-envoy\") pod \"555228e6-5a19-4924-a121-9e4d321c0104\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") "
Oct 30 19:17:09.780330 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.780198    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-istio-data\") pod \"555228e6-5a19-4924-a121-9e4d321c0104\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") "
Oct 30 19:17:09.780330 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.780257    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/555228e6-5a19-4924-a121-9e4d321c0104-kube-api-access-tkc2t\" (UniqueName: \"kubernetes.io/projected/555228e6-5a19-4924-a121-9e4d321c0104-kube-api-access-tkc2t\") pod \"555228e6-5a19-4924-a121-9e4d321c0104\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") "
Oct 30 19:17:09.780330 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.780300    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-token\" (UniqueName: \"kubernetes.io/projected/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-token\") pod \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") "
Oct 30 19:17:09.780575 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.780357    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/555228e6-5a19-4924-a121-9e4d321c0104-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/555228e6-5a19-4924-a121-9e4d321c0104-istiod-ca-cert\") pod \"555228e6-5a19-4924-a121-9e4d321c0104\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") "
Oct 30 19:17:09.780575 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.780379    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-kube-api-access-v8dfj\" (UniqueName: \"kubernetes.io/projected/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-kube-api-access-v8dfj\") pod \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") "
Oct 30 19:17:09.780575 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.780395    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-credential-socket" pod "34ce2d7c-43f1-41cc-9fb4-2f2e058a4609" (UID: "34ce2d7c-43f1-41cc-9fb4-2f2e058a4609"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:09.780575 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.780402    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-workload-socket\") pod \"555228e6-5a19-4924-a121-9e4d321c0104\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") "
Oct 30 19:17:09.780575 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.780492    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-workload-socket\") pod \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") "
Oct 30 19:17:09.780575 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.780551    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-workload-certs\") pod \"555228e6-5a19-4924-a121-9e4d321c0104\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") "
Oct 30 19:17:09.780888 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.780587    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/555228e6-5a19-4924-a121-9e4d321c0104-istio-token\" (UniqueName: \"kubernetes.io/projected/555228e6-5a19-4924-a121-9e4d321c0104-istio-token\") pod \"555228e6-5a19-4924-a121-9e4d321c0104\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") "
Oct 30 19:17:09.780888 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.780618    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-workload-certs\") pod \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") "
Oct 30 19:17:09.780888 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.780630    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-workload-socket" pod "555228e6-5a19-4924-a121-9e4d321c0104" (UID: "555228e6-5a19-4924-a121-9e4d321c0104"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:09.780888 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.780674    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-podinfo\") pod \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") "
Oct 30 19:17:09.780888 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.780712    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-credential-socket\") pod \"555228e6-5a19-4924-a121-9e4d321c0104\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") "
Oct 30 19:17:09.780888 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.780749    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/555228e6-5a19-4924-a121-9e4d321c0104-istiod-ca-cert" pod "555228e6-5a19-4924-a121-9e4d321c0104" (UID: "555228e6-5a19-4924-a121-9e4d321c0104"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:09.780888 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.780763    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-envoy\") pod \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") "
Oct 30 19:17:09.780888 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.780858    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-workload-certs" pod "555228e6-5a19-4924-a121-9e4d321c0104" (UID: "555228e6-5a19-4924-a121-9e4d321c0104"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:09.780888 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.780878    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istiod-ca-cert\") pod \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") "
Oct 30 19:17:09.780888 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.780879    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-istio-data" pod "555228e6-5a19-4924-a121-9e4d321c0104" (UID: "555228e6-5a19-4924-a121-9e4d321c0104"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:09.781481 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.780924    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/555228e6-5a19-4924-a121-9e4d321c0104-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/555228e6-5a19-4924-a121-9e4d321c0104-istio-podinfo\") pod \"555228e6-5a19-4924-a121-9e4d321c0104\" (UID: \"555228e6-5a19-4924-a121-9e4d321c0104\") "
Oct 30 19:17:09.781481 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.780964    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-data\") pod \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\" (UID: \"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609\") "
Oct 30 19:17:09.781481 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.781084    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-workload-socket" pod "34ce2d7c-43f1-41cc-9fb4-2f2e058a4609" (UID: "34ce2d7c-43f1-41cc-9fb4-2f2e058a4609"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:09.781481 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.781166    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-workload-certs" pod "34ce2d7c-43f1-41cc-9fb4-2f2e058a4609" (UID: "34ce2d7c-43f1-41cc-9fb4-2f2e058a4609"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:09.781481 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.781445    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istiod-ca-cert" pod "34ce2d7c-43f1-41cc-9fb4-2f2e058a4609" (UID: "34ce2d7c-43f1-41cc-9fb4-2f2e058a4609"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:09.781733 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.781648    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-credential-socket" pod "555228e6-5a19-4924-a121-9e4d321c0104" (UID: "555228e6-5a19-4924-a121-9e4d321c0104"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:09.781733 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.781685    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.781733 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.781704    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.781733 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.781720    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.781733 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.781733    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/371f8b40-cf29-431e-be68-d3c7ce0eab02-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.781979 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.781747    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.781979 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.781761    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/555228e6-5a19-4924-a121-9e4d321c0104-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.781979 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.781776    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.781979 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.781793    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.781979 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.781808    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.781979 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.781822    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.781979 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.781837    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghhkq\" (UniqueName: \"kubernetes.io/projected/371f8b40-cf29-431e-be68-d3c7ce0eab02-kube-api-access-ghhkq\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.781979 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.781852    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.781979 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.781868    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.781979 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.781883    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/371f8b40-cf29-431e-be68-d3c7ce0eab02-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.781979 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.781923    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.782586 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.781404    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-data" pod "34ce2d7c-43f1-41cc-9fb4-2f2e058a4609" (UID: "34ce2d7c-43f1-41cc-9fb4-2f2e058a4609"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:09.783338 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.783311    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-envoy" pod "34ce2d7c-43f1-41cc-9fb4-2f2e058a4609" (UID: "34ce2d7c-43f1-41cc-9fb4-2f2e058a4609"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:09.783466 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.783315    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-istio-envoy" pod "555228e6-5a19-4924-a121-9e4d321c0104" (UID: "555228e6-5a19-4924-a121-9e4d321c0104"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:09.783654 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.783628    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-podinfo" pod "34ce2d7c-43f1-41cc-9fb4-2f2e058a4609" (UID: "34ce2d7c-43f1-41cc-9fb4-2f2e058a4609"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:09.783848 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.783828    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-token" pod "34ce2d7c-43f1-41cc-9fb4-2f2e058a4609" (UID: "34ce2d7c-43f1-41cc-9fb4-2f2e058a4609"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:09.784174 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.784146    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555228e6-5a19-4924-a121-9e4d321c0104-kube-api-access-tkc2t" pod "555228e6-5a19-4924-a121-9e4d321c0104" (UID: "555228e6-5a19-4924-a121-9e4d321c0104"). InnerVolumeSpecName "kube-api-access-tkc2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:09.784539 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.784519    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/555228e6-5a19-4924-a121-9e4d321c0104-istio-podinfo" pod "555228e6-5a19-4924-a121-9e4d321c0104" (UID: "555228e6-5a19-4924-a121-9e4d321c0104"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:09.784623 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.784519    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-kube-api-access-v8dfj" pod "34ce2d7c-43f1-41cc-9fb4-2f2e058a4609" (UID: "34ce2d7c-43f1-41cc-9fb4-2f2e058a4609"). InnerVolumeSpecName "kube-api-access-v8dfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:09.784938 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.784917    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555228e6-5a19-4924-a121-9e4d321c0104-istio-token" pod "555228e6-5a19-4924-a121-9e4d321c0104" (UID: "555228e6-5a19-4924-a121-9e4d321c0104"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:09.825429 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.825382    4924 generic.go:356] "Generic (PLEG): container finished" podID="34ce2d7c-43f1-41cc-9fb4-2f2e058a4609" containerID="a0507c3c12405c1dc1082bc99c2fc7b345b63590d6056dbaff0873cb5512c70a" exitCode=0
Oct 30 19:17:09.825621 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.825435    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"
Oct 30 19:17:09.825621 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.825436    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4" event={"ID":"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609","Type":"ContainerDied","Data":"a0507c3c12405c1dc1082bc99c2fc7b345b63590d6056dbaff0873cb5512c70a"}
Oct 30 19:17:09.825621 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.825506    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4" event={"ID":"34ce2d7c-43f1-41cc-9fb4-2f2e058a4609","Type":"ContainerDied","Data":"97b32ef90853215428b5317e283e004d6d19bbdcf223467a0e3ed06c7db5bad3"}
Oct 30 19:17:09.825621 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.825536    4924 scope.go:122] "RemoveContainer" containerID="a0507c3c12405c1dc1082bc99c2fc7b345b63590d6056dbaff0873cb5512c70a"
Oct 30 19:17:09.828284 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.828261    4924 generic.go:356] "Generic (PLEG): container finished" podID="371f8b40-cf29-431e-be68-d3c7ce0eab02" containerID="1818fe2ebe802940fb40d3282c59ab94189fd7cd225e0666a90047646f79485c" exitCode=0
Oct 30 19:17:09.828412 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.828341    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9" event={"ID":"371f8b40-cf29-431e-be68-d3c7ce0eab02","Type":"ContainerDied","Data":"1818fe2ebe802940fb40d3282c59ab94189fd7cd225e0666a90047646f79485c"}
Oct 30 19:17:09.828412 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.828383    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9" event={"ID":"371f8b40-cf29-431e-be68-d3c7ce0eab02","Type":"ContainerDied","Data":"5d87658921e1e64a1d39afa38fd51cceabc5aebe273158332dd23d7f3f1d8642"}
Oct 30 19:17:09.828412 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.828344    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"
Oct 30 19:17:09.829782 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.829759    4924 generic.go:356] "Generic (PLEG): container finished" podID="555228e6-5a19-4924-a121-9e4d321c0104" containerID="fc019e14612d4637ca7db4f3e772ec5ee0981722a41da1979884e155d5c85e6e" exitCode=0
Oct 30 19:17:09.829904 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.829784    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw" event={"ID":"555228e6-5a19-4924-a121-9e4d321c0104","Type":"ContainerDied","Data":"fc019e14612d4637ca7db4f3e772ec5ee0981722a41da1979884e155d5c85e6e"}
Oct 30 19:17:09.829904 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.829807    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"
Oct 30 19:17:09.829904 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.829825    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw" event={"ID":"555228e6-5a19-4924-a121-9e4d321c0104","Type":"ContainerDied","Data":"2a72e771036fc36d0217fc3395d5aa9afab7e6ffff2f8cc449ce808d69a14d8f"}
Oct 30 19:17:09.839712 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.839695    4924 scope.go:122] "RemoveContainer" containerID="a0507c3c12405c1dc1082bc99c2fc7b345b63590d6056dbaff0873cb5512c70a"
Oct 30 19:17:09.840045 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:17:09.840016    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0507c3c12405c1dc1082bc99c2fc7b345b63590d6056dbaff0873cb5512c70a\": container with ID starting with a0507c3c12405c1dc1082bc99c2fc7b345b63590d6056dbaff0873cb5512c70a not found: ID does not exist" containerID="a0507c3c12405c1dc1082bc99c2fc7b345b63590d6056dbaff0873cb5512c70a"
Oct 30 19:17:09.840307 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.840057    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0507c3c12405c1dc1082bc99c2fc7b345b63590d6056dbaff0873cb5512c70a"} err="failed to get container status \"a0507c3c12405c1dc1082bc99c2fc7b345b63590d6056dbaff0873cb5512c70a\": rpc error: code = NotFound desc = could not find container \"a0507c3c12405c1dc1082bc99c2fc7b345b63590d6056dbaff0873cb5512c70a\": container with ID starting with a0507c3c12405c1dc1082bc99c2fc7b345b63590d6056dbaff0873cb5512c70a not found: ID does not exist"
Oct 30 19:17:09.840307 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.840114    4924 scope.go:122] "RemoveContainer" containerID="1818fe2ebe802940fb40d3282c59ab94189fd7cd225e0666a90047646f79485c"
Oct 30 19:17:09.852308 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.852288    4924 scope.go:122] "RemoveContainer" containerID="1818fe2ebe802940fb40d3282c59ab94189fd7cd225e0666a90047646f79485c"
Oct 30 19:17:09.852587 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:17:09.852569    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1818fe2ebe802940fb40d3282c59ab94189fd7cd225e0666a90047646f79485c\": container with ID starting with 1818fe2ebe802940fb40d3282c59ab94189fd7cd225e0666a90047646f79485c not found: ID does not exist" containerID="1818fe2ebe802940fb40d3282c59ab94189fd7cd225e0666a90047646f79485c"
Oct 30 19:17:09.852657 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.852595    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1818fe2ebe802940fb40d3282c59ab94189fd7cd225e0666a90047646f79485c"} err="failed to get container status \"1818fe2ebe802940fb40d3282c59ab94189fd7cd225e0666a90047646f79485c\": rpc error: code = NotFound desc = could not find container \"1818fe2ebe802940fb40d3282c59ab94189fd7cd225e0666a90047646f79485c\": container with ID starting with 1818fe2ebe802940fb40d3282c59ab94189fd7cd225e0666a90047646f79485c not found: ID does not exist"
Oct 30 19:17:09.852657 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.852612    4924 scope.go:122] "RemoveContainer" containerID="fc019e14612d4637ca7db4f3e772ec5ee0981722a41da1979884e155d5c85e6e"
Oct 30 19:17:09.862343 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.862321    4924 scope.go:122] "RemoveContainer" containerID="fc019e14612d4637ca7db4f3e772ec5ee0981722a41da1979884e155d5c85e6e"
Oct 30 19:17:09.862597 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:17:09.862577    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc019e14612d4637ca7db4f3e772ec5ee0981722a41da1979884e155d5c85e6e\": container with ID starting with fc019e14612d4637ca7db4f3e772ec5ee0981722a41da1979884e155d5c85e6e not found: ID does not exist" containerID="fc019e14612d4637ca7db4f3e772ec5ee0981722a41da1979884e155d5c85e6e"
Oct 30 19:17:09.862658 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.862608    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc019e14612d4637ca7db4f3e772ec5ee0981722a41da1979884e155d5c85e6e"} err="failed to get container status \"fc019e14612d4637ca7db4f3e772ec5ee0981722a41da1979884e155d5c85e6e\": rpc error: code = NotFound desc = could not find container \"fc019e14612d4637ca7db4f3e772ec5ee0981722a41da1979884e155d5c85e6e\": container with ID starting with fc019e14612d4637ca7db4f3e772ec5ee0981722a41da1979884e155d5c85e6e not found: ID does not exist"
Oct 30 19:17:09.882669 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.882641    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkc2t\" (UniqueName: \"kubernetes.io/projected/555228e6-5a19-4924-a121-9e4d321c0104-kube-api-access-tkc2t\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.882669 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.882669    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.882832 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.882684    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8dfj\" (UniqueName: \"kubernetes.io/projected/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-kube-api-access-v8dfj\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.882832 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.882697    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/555228e6-5a19-4924-a121-9e4d321c0104-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.882832 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.882708    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.882832 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.882722    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.882832 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.882736    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.882832 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.882749    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/555228e6-5a19-4924-a121-9e4d321c0104-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.882832 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.882761    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:09.882832 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:09.882773    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/555228e6-5a19-4924-a121-9e4d321c0104-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:10.118868 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:10.118787    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"]
Oct 30 19:17:10.124002 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:10.123974    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-qtlm9"]
Oct 30 19:17:10.228726 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:10.228694    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"]
Oct 30 19:17:10.233771 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:10.233746    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-7zwt4"]
Oct 30 19:17:10.337022 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:10.336988    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"]
Oct 30 19:17:10.339683 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:10.339658    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-314bd6fa-openshift-default-69d8f74b5c-zp9kw"]
Oct 30 19:17:10.417297 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:10.417206    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d" podUID="b9cd1a58-ecd4-4611-aaac-28109cd2981a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:17:10.417297 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:10.417236    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf" podUID="dc3baffc-49c4-4101-9bf4-047229d89000" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:17:10.622626 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:10.622589    4924 patch_prober.go:29] interesting pod/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:17:10.622802 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:10.622653    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w" podUID="d6ec64bd-1810-4f78-b451-9faae648be6e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:17:10.781437 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:10.781359    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:10.834630 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:10.834598    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34ce2d7c-43f1-41cc-9fb4-2f2e058a4609" path="/var/lib/kubelet/pods/34ce2d7c-43f1-41cc-9fb4-2f2e058a4609/volumes"
Oct 30 19:17:10.835402 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:10.835382    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="371f8b40-cf29-431e-be68-d3c7ce0eab02" path="/var/lib/kubelet/pods/371f8b40-cf29-431e-be68-d3c7ce0eab02/volumes"
Oct 30 19:17:10.835980 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:10.835964    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="555228e6-5a19-4924-a121-9e4d321c0104" path="/var/lib/kubelet/pods/555228e6-5a19-4924-a121-9e4d321c0104/volumes"
Oct 30 19:17:11.520489 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.520457    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:11.636155 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.636117    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a41e9e11-9wrkt"]
Oct 30 19:17:11.636596 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.636583    4924 cpu_manager.go:414] "containerMap: removing container" podUID="371f8b40-cf29-431e-be68-d3c7ce0eab02" containerName="istio-proxy"
Oct 30 19:17:11.636639 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.636597    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="371f8b40-cf29-431e-be68-d3c7ce0eab02" containerName="istio-proxy"
Oct 30 19:17:11.636639 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.636609    4924 cpu_manager.go:414] "containerMap: removing container" podUID="0580aebc-4e9e-4578-872c-943719ee559f" containerName="payload-processing"
Oct 30 19:17:11.636639 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.636614    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0580aebc-4e9e-4578-872c-943719ee559f" containerName="payload-processing"
Oct 30 19:17:11.636639 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.636620    4924 cpu_manager.go:414] "containerMap: removing container" podUID="04617682-2d5b-421c-82c4-e7e60c01edbf" containerName="revoke-keys"
Oct 30 19:17:11.636639 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.636627    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04617682-2d5b-421c-82c4-e7e60c01edbf" containerName="revoke-keys"
Oct 30 19:17:11.636817 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.636642    4924 cpu_manager.go:414] "containerMap: removing container" podUID="d3dd0455-39e6-4b1b-80ee-ffbde02d4f00" containerName="payload-pre-processing"
Oct 30 19:17:11.636817 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.636646    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3dd0455-39e6-4b1b-80ee-ffbde02d4f00" containerName="payload-pre-processing"
Oct 30 19:17:11.636817 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.636657    4924 cpu_manager.go:414] "containerMap: removing container" podUID="34ce2d7c-43f1-41cc-9fb4-2f2e058a4609" containerName="istio-proxy"
Oct 30 19:17:11.636817 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.636663    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34ce2d7c-43f1-41cc-9fb4-2f2e058a4609" containerName="istio-proxy"
Oct 30 19:17:11.636817 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.636675    4924 cpu_manager.go:414] "containerMap: removing container" podUID="96230ce5-7a7e-48f9-a87f-1b786d7f8730" containerName="revoke-keys"
Oct 30 19:17:11.636817 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.636680    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96230ce5-7a7e-48f9-a87f-1b786d7f8730" containerName="revoke-keys"
Oct 30 19:17:11.636817 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.636696    4924 cpu_manager.go:414] "containerMap: removing container" podUID="555228e6-5a19-4924-a121-9e4d321c0104" containerName="istio-proxy"
Oct 30 19:17:11.636817 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.636702    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="555228e6-5a19-4924-a121-9e4d321c0104" containerName="istio-proxy"
Oct 30 19:17:11.636817 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.636775    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="555228e6-5a19-4924-a121-9e4d321c0104" containerName="istio-proxy"
Oct 30 19:17:11.636817 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.636783    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="371f8b40-cf29-431e-be68-d3c7ce0eab02" containerName="istio-proxy"
Oct 30 19:17:11.636817 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.636789    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="0580aebc-4e9e-4578-872c-943719ee559f" containerName="payload-processing"
Oct 30 19:17:11.636817 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.636796    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="04617682-2d5b-421c-82c4-e7e60c01edbf" containerName="revoke-keys"
Oct 30 19:17:11.636817 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.636803    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3dd0455-39e6-4b1b-80ee-ffbde02d4f00" containerName="payload-pre-processing"
Oct 30 19:17:11.636817 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.636810    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="34ce2d7c-43f1-41cc-9fb4-2f2e058a4609" containerName="istio-proxy"
Oct 30 19:17:11.636817 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.636817    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="96230ce5-7a7e-48f9-a87f-1b786d7f8730" containerName="revoke-keys"
Oct 30 19:17:11.641653 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.641632    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41e9e11-9wrkt"
Oct 30 19:17:11.693796 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.693766    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:17:11.696896 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.696865    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a41e9e11-9wrkt"]
Oct 30 19:17:11.703684 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.703650    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/996da56a-ede7-4041-b0c4-1d5f10f1b659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a41e9e11-9wrkt\" (UID: \"996da56a-ede7-4041-b0c4-1d5f10f1b659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41e9e11-9wrkt"
Oct 30 19:17:11.703832 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.703812    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/996da56a-ede7-4041-b0c4-1d5f10f1b659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a41e9e11-9wrkt\" (UID: \"996da56a-ede7-4041-b0c4-1d5f10f1b659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41e9e11-9wrkt"
Oct 30 19:17:11.703901 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.703848    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzt4m\" (UniqueName: \"kubernetes.io/projected/996da56a-ede7-4041-b0c4-1d5f10f1b659-kube-api-access-xzt4m\") pod \"maas-api-revoke-sub-a41e9e11-9wrkt\" (UID: \"996da56a-ede7-4041-b0c4-1d5f10f1b659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41e9e11-9wrkt"
Oct 30 19:17:11.740293 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.740270    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:17:11.740432 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.740310    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:17:11.740432 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.740394    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:17:11.804701 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.804626    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/996da56a-ede7-4041-b0c4-1d5f10f1b659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a41e9e11-9wrkt\" (UID: \"996da56a-ede7-4041-b0c4-1d5f10f1b659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41e9e11-9wrkt"
Oct 30 19:17:11.804701 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.804660    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzt4m\" (UniqueName: \"kubernetes.io/projected/996da56a-ede7-4041-b0c4-1d5f10f1b659-kube-api-access-xzt4m\") pod \"maas-api-revoke-sub-a41e9e11-9wrkt\" (UID: \"996da56a-ede7-4041-b0c4-1d5f10f1b659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41e9e11-9wrkt"
Oct 30 19:17:11.804896 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.804762    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/996da56a-ede7-4041-b0c4-1d5f10f1b659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a41e9e11-9wrkt\" (UID: \"996da56a-ede7-4041-b0c4-1d5f10f1b659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41e9e11-9wrkt"
Oct 30 19:17:11.805390 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.805368    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/996da56a-ede7-4041-b0c4-1d5f10f1b659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a41e9e11-9wrkt\" (UID: \"996da56a-ede7-4041-b0c4-1d5f10f1b659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41e9e11-9wrkt"
Oct 30 19:17:11.806966 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.806946    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/996da56a-ede7-4041-b0c4-1d5f10f1b659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a41e9e11-9wrkt\" (UID: \"996da56a-ede7-4041-b0c4-1d5f10f1b659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41e9e11-9wrkt"
Oct 30 19:17:11.861891 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.861820    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzt4m\" (UniqueName: \"kubernetes.io/projected/996da56a-ede7-4041-b0c4-1d5f10f1b659-kube-api-access-xzt4m\") pod \"maas-api-revoke-sub-a41e9e11-9wrkt\" (UID: \"996da56a-ede7-4041-b0c4-1d5f10f1b659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41e9e11-9wrkt"
Oct 30 19:17:11.955536 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.955507    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41e9e11-9wrkt"
Oct 30 19:17:11.963821 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:11.963799    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:12.330111 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:12.326428    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a41e9e11-9wrkt"]
Oct 30 19:17:12.856408 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:12.856365    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41e9e11-9wrkt" event={"ID":"996da56a-ede7-4041-b0c4-1d5f10f1b659","Type":"ContainerStarted","Data":"4f8684a0805b765c383cb0f1d4494990c0bf44389070185beca00c85a51eb158"}
Oct 30 19:17:12.856782 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:12.856431    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41e9e11-9wrkt" event={"ID":"996da56a-ede7-4041-b0c4-1d5f10f1b659","Type":"ContainerStarted","Data":"d271d045ba75f4fff3eef5be9e263153ce2d322b86eb9fc7ba97ee9b8d0122fb"}
Oct 30 19:17:12.969039 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:12.968931    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41e9e11-9wrkt" podStartSLOduration=1.968917118 podStartE2EDuration="1.968917118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:12.966756925 +0000 UTC m=+2434.243807375" watchObservedRunningTime="2026-10-30 19:17:12.968917118 +0000 UTC m=+2434.245967562"
Oct 30 19:17:13.864724 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:13.864690    4924 generic.go:356] "Generic (PLEG): container finished" podID="996da56a-ede7-4041-b0c4-1d5f10f1b659" containerID="4f8684a0805b765c383cb0f1d4494990c0bf44389070185beca00c85a51eb158" exitCode=0
Oct 30 19:17:13.865215 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:13.864732    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41e9e11-9wrkt" event={"ID":"996da56a-ede7-4041-b0c4-1d5f10f1b659","Type":"ContainerDied","Data":"4f8684a0805b765c383cb0f1d4494990c0bf44389070185beca00c85a51eb158"}
Oct 30 19:17:14.681623 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.681593    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 19:17:14.687822 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.687800    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d"
Oct 30 19:17:14.740119 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.740048    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6ec64bd-1810-4f78-b451-9faae648be6e-kube-api-access-wz4cd\" (UniqueName: \"kubernetes.io/projected/d6ec64bd-1810-4f78-b451-9faae648be6e-kube-api-access-wz4cd\") pod \"d6ec64bd-1810-4f78-b451-9faae648be6e\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") "
Oct 30 19:17:14.740309 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.740129    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-envoy\") pod \"d6ec64bd-1810-4f78-b451-9faae648be6e\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") "
Oct 30 19:17:14.740309 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.740222    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6ec64bd-1810-4f78-b451-9faae648be6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6ec64bd-1810-4f78-b451-9faae648be6e-istiod-ca-cert\") pod \"d6ec64bd-1810-4f78-b451-9faae648be6e\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") "
Oct 30 19:17:14.740309 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.740285    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-credential-socket\") pod \"d6ec64bd-1810-4f78-b451-9faae648be6e\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") "
Oct 30 19:17:14.740500 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.740347    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9cd1a58-ecd4-4611-aaac-28109cd2981a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9cd1a58-ecd4-4611-aaac-28109cd2981a-plugins-config-volume\") pod \"b9cd1a58-ecd4-4611-aaac-28109cd2981a\" (UID: \"b9cd1a58-ecd4-4611-aaac-28109cd2981a\") "
Oct 30 19:17:14.740500 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.740418    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-workload-certs\") pod \"d6ec64bd-1810-4f78-b451-9faae648be6e\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") "
Oct 30 19:17:14.740627 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.740512    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-workload-socket\") pod \"d6ec64bd-1810-4f78-b451-9faae648be6e\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") "
Oct 30 19:17:14.740627 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.740537    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-credential-socket" pod "d6ec64bd-1810-4f78-b451-9faae648be6e" (UID: "d6ec64bd-1810-4f78-b451-9faae648be6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.740627 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.740564    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-podinfo\") pod \"d6ec64bd-1810-4f78-b451-9faae648be6e\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") "
Oct 30 19:17:14.740831 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.740628    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9cd1a58-ecd4-4611-aaac-28109cd2981a-kube-api-access-xs7mk\" (UniqueName: \"kubernetes.io/projected/b9cd1a58-ecd4-4611-aaac-28109cd2981a-kube-api-access-xs7mk\") pod \"b9cd1a58-ecd4-4611-aaac-28109cd2981a\" (UID: \"b9cd1a58-ecd4-4611-aaac-28109cd2981a\") "
Oct 30 19:17:14.740831 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.740635    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6ec64bd-1810-4f78-b451-9faae648be6e-istiod-ca-cert" pod "d6ec64bd-1810-4f78-b451-9faae648be6e" (UID: "d6ec64bd-1810-4f78-b451-9faae648be6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:14.740831 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.740739    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9cd1a58-ecd4-4611-aaac-28109cd2981a-plugins-config-volume" pod "b9cd1a58-ecd4-4611-aaac-28109cd2981a" (UID: "b9cd1a58-ecd4-4611-aaac-28109cd2981a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:14.740831 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.740744    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-token\" (UniqueName: \"kubernetes.io/projected/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-token\") pod \"d6ec64bd-1810-4f78-b451-9faae648be6e\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") "
Oct 30 19:17:14.740831 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.740789    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-data\") pod \"d6ec64bd-1810-4f78-b451-9faae648be6e\" (UID: \"d6ec64bd-1810-4f78-b451-9faae648be6e\") "
Oct 30 19:17:14.741244 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.740943    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-workload-socket" pod "d6ec64bd-1810-4f78-b451-9faae648be6e" (UID: "d6ec64bd-1810-4f78-b451-9faae648be6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.741244 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.741007    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-workload-certs" pod "d6ec64bd-1810-4f78-b451-9faae648be6e" (UID: "d6ec64bd-1810-4f78-b451-9faae648be6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.741244 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.741175    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-data" pod "d6ec64bd-1810-4f78-b451-9faae648be6e" (UID: "d6ec64bd-1810-4f78-b451-9faae648be6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.741435 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.741354    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.741435 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.741377    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6ec64bd-1810-4f78-b451-9faae648be6e-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.741435 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.741392    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.741435 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.741409    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9cd1a58-ecd4-4611-aaac-28109cd2981a-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.741435 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.741433    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.741691 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.741446    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.742828 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.742800    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-envoy" pod "d6ec64bd-1810-4f78-b451-9faae648be6e" (UID: "d6ec64bd-1810-4f78-b451-9faae648be6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:14.743092 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.743047    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-podinfo" pod "d6ec64bd-1810-4f78-b451-9faae648be6e" (UID: "d6ec64bd-1810-4f78-b451-9faae648be6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:14.743200 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.743178    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-token" pod "d6ec64bd-1810-4f78-b451-9faae648be6e" (UID: "d6ec64bd-1810-4f78-b451-9faae648be6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:14.743387 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.743364    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ec64bd-1810-4f78-b451-9faae648be6e-kube-api-access-wz4cd" pod "d6ec64bd-1810-4f78-b451-9faae648be6e" (UID: "d6ec64bd-1810-4f78-b451-9faae648be6e"). InnerVolumeSpecName "kube-api-access-wz4cd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:14.744520 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.744498    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9cd1a58-ecd4-4611-aaac-28109cd2981a-kube-api-access-xs7mk" pod "b9cd1a58-ecd4-4611-aaac-28109cd2981a" (UID: "b9cd1a58-ecd4-4611-aaac-28109cd2981a"). InnerVolumeSpecName "kube-api-access-xs7mk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:14.752984 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.752965    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf"
Oct 30 19:17:14.841913 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.841871    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc3baffc-49c4-4101-9bf4-047229d89000-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3baffc-49c4-4101-9bf4-047229d89000-plugins-config-volume\") pod \"dc3baffc-49c4-4101-9bf4-047229d89000\" (UID: \"dc3baffc-49c4-4101-9bf4-047229d89000\") "
Oct 30 19:17:14.842151 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.842035    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc3baffc-49c4-4101-9bf4-047229d89000-kube-api-access-j84j7\" (UniqueName: \"kubernetes.io/projected/dc3baffc-49c4-4101-9bf4-047229d89000-kube-api-access-j84j7\") pod \"dc3baffc-49c4-4101-9bf4-047229d89000\" (UID: \"dc3baffc-49c4-4101-9bf4-047229d89000\") "
Oct 30 19:17:14.842403 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.842378    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc3baffc-49c4-4101-9bf4-047229d89000-plugins-config-volume" pod "dc3baffc-49c4-4101-9bf4-047229d89000" (UID: "dc3baffc-49c4-4101-9bf4-047229d89000"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:14.842511 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.842488    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.842570 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.842517    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs7mk\" (UniqueName: \"kubernetes.io/projected/b9cd1a58-ecd4-4611-aaac-28109cd2981a-kube-api-access-xs7mk\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.842570 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.842531    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.842570 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.842545    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz4cd\" (UniqueName: \"kubernetes.io/projected/d6ec64bd-1810-4f78-b451-9faae648be6e-kube-api-access-wz4cd\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.842570 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.842558    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6ec64bd-1810-4f78-b451-9faae648be6e-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.842781 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.842570    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3baffc-49c4-4101-9bf4-047229d89000-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.848344 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.848312    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc3baffc-49c4-4101-9bf4-047229d89000-kube-api-access-j84j7" pod "dc3baffc-49c4-4101-9bf4-047229d89000" (UID: "dc3baffc-49c4-4101-9bf4-047229d89000"). InnerVolumeSpecName "kube-api-access-j84j7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:14.875602 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.875567    4924 generic.go:356] "Generic (PLEG): container finished" podID="dc3baffc-49c4-4101-9bf4-047229d89000" containerID="922c79e8290ed4f53737c6383a853630095dcff3ea014c272638e4fdece41ef1" exitCode=0
Oct 30 19:17:14.876165 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.875629    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf"
Oct 30 19:17:14.876165 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.875656    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf" event={"ID":"dc3baffc-49c4-4101-9bf4-047229d89000","Type":"ContainerDied","Data":"922c79e8290ed4f53737c6383a853630095dcff3ea014c272638e4fdece41ef1"}
Oct 30 19:17:14.876165 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.875741    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf" event={"ID":"dc3baffc-49c4-4101-9bf4-047229d89000","Type":"ContainerDied","Data":"b0996816d7d5b5bcc729cf9b0ab7f749fc9a889324348f968452269ad9eeb238"}
Oct 30 19:17:14.876165 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.875776    4924 scope.go:122] "RemoveContainer" containerID="922c79e8290ed4f53737c6383a853630095dcff3ea014c272638e4fdece41ef1"
Oct 30 19:17:14.878119 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.878094    4924 generic.go:356] "Generic (PLEG): container finished" podID="d6ec64bd-1810-4f78-b451-9faae648be6e" containerID="46ba7f99d80df3f1099875ba25165476edd2516c73f2a949109eb012a3423062" exitCode=0
Oct 30 19:17:14.878219 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.878154    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"
Oct 30 19:17:14.878291 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.878249    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w" event={"ID":"d6ec64bd-1810-4f78-b451-9faae648be6e","Type":"ContainerDied","Data":"46ba7f99d80df3f1099875ba25165476edd2516c73f2a949109eb012a3423062"}
Oct 30 19:17:14.878351 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.878303    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w" event={"ID":"d6ec64bd-1810-4f78-b451-9faae648be6e","Type":"ContainerDied","Data":"ccb204b15c58d4a332d812d15466088da075a18345ffa09daa590018cbb0f791"}
Oct 30 19:17:14.880126 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.880102    4924 generic.go:356] "Generic (PLEG): container finished" podID="b9cd1a58-ecd4-4611-aaac-28109cd2981a" containerID="2164deca5b6f9cbd971790debf5d890e560ef56d52597679275771209c898bc1" exitCode=0
Oct 30 19:17:14.880258 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.880157    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d"
Oct 30 19:17:14.880258 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.880199    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d" event={"ID":"b9cd1a58-ecd4-4611-aaac-28109cd2981a","Type":"ContainerDied","Data":"2164deca5b6f9cbd971790debf5d890e560ef56d52597679275771209c898bc1"}
Oct 30 19:17:14.880258 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.880235    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d" event={"ID":"b9cd1a58-ecd4-4611-aaac-28109cd2981a","Type":"ContainerDied","Data":"4a295b0d3e01c320b372484224f5d85e962c642c36f128c5f0e0b5ca4799d47f"}
Oct 30 19:17:14.889642 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.889616    4924 scope.go:122] "RemoveContainer" containerID="922c79e8290ed4f53737c6383a853630095dcff3ea014c272638e4fdece41ef1"
Oct 30 19:17:14.889966 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:17:14.889942    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"922c79e8290ed4f53737c6383a853630095dcff3ea014c272638e4fdece41ef1\": container with ID starting with 922c79e8290ed4f53737c6383a853630095dcff3ea014c272638e4fdece41ef1 not found: ID does not exist" containerID="922c79e8290ed4f53737c6383a853630095dcff3ea014c272638e4fdece41ef1"
Oct 30 19:17:14.890081 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.889982    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"922c79e8290ed4f53737c6383a853630095dcff3ea014c272638e4fdece41ef1"} err="failed to get container status \"922c79e8290ed4f53737c6383a853630095dcff3ea014c272638e4fdece41ef1\": rpc error: code = NotFound desc = could not find container \"922c79e8290ed4f53737c6383a853630095dcff3ea014c272638e4fdece41ef1\": container with ID starting with 922c79e8290ed4f53737c6383a853630095dcff3ea014c272638e4fdece41ef1 not found: ID does not exist"
Oct 30 19:17:14.890081 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.890002    4924 scope.go:122] "RemoveContainer" containerID="46ba7f99d80df3f1099875ba25165476edd2516c73f2a949109eb012a3423062"
Oct 30 19:17:14.900040 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.900020    4924 scope.go:122] "RemoveContainer" containerID="8368c268d84e8ae8b1654db113e7462130487deb364760b85708c6aa49205751"
Oct 30 19:17:14.911988 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.911971    4924 scope.go:122] "RemoveContainer" containerID="46ba7f99d80df3f1099875ba25165476edd2516c73f2a949109eb012a3423062"
Oct 30 19:17:14.912260 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:17:14.912241    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46ba7f99d80df3f1099875ba25165476edd2516c73f2a949109eb012a3423062\": container with ID starting with 46ba7f99d80df3f1099875ba25165476edd2516c73f2a949109eb012a3423062 not found: ID does not exist" containerID="46ba7f99d80df3f1099875ba25165476edd2516c73f2a949109eb012a3423062"
Oct 30 19:17:14.912359 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.912267    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ba7f99d80df3f1099875ba25165476edd2516c73f2a949109eb012a3423062"} err="failed to get container status \"46ba7f99d80df3f1099875ba25165476edd2516c73f2a949109eb012a3423062\": rpc error: code = NotFound desc = could not find container \"46ba7f99d80df3f1099875ba25165476edd2516c73f2a949109eb012a3423062\": container with ID starting with 46ba7f99d80df3f1099875ba25165476edd2516c73f2a949109eb012a3423062 not found: ID does not exist"
Oct 30 19:17:14.912359 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.912284    4924 scope.go:122] "RemoveContainer" containerID="8368c268d84e8ae8b1654db113e7462130487deb364760b85708c6aa49205751"
Oct 30 19:17:14.912549 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:17:14.912533    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8368c268d84e8ae8b1654db113e7462130487deb364760b85708c6aa49205751\": container with ID starting with 8368c268d84e8ae8b1654db113e7462130487deb364760b85708c6aa49205751 not found: ID does not exist" containerID="8368c268d84e8ae8b1654db113e7462130487deb364760b85708c6aa49205751"
Oct 30 19:17:14.912600 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.912556    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8368c268d84e8ae8b1654db113e7462130487deb364760b85708c6aa49205751"} err="failed to get container status \"8368c268d84e8ae8b1654db113e7462130487deb364760b85708c6aa49205751\": rpc error: code = NotFound desc = could not find container \"8368c268d84e8ae8b1654db113e7462130487deb364760b85708c6aa49205751\": container with ID starting with 8368c268d84e8ae8b1654db113e7462130487deb364760b85708c6aa49205751 not found: ID does not exist"
Oct 30 19:17:14.912600 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.912572    4924 scope.go:122] "RemoveContainer" containerID="2164deca5b6f9cbd971790debf5d890e560ef56d52597679275771209c898bc1"
Oct 30 19:17:14.923590 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.923563    4924 scope.go:122] "RemoveContainer" containerID="2164deca5b6f9cbd971790debf5d890e560ef56d52597679275771209c898bc1"
Oct 30 19:17:14.923897 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:17:14.923870    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2164deca5b6f9cbd971790debf5d890e560ef56d52597679275771209c898bc1\": container with ID starting with 2164deca5b6f9cbd971790debf5d890e560ef56d52597679275771209c898bc1 not found: ID does not exist" containerID="2164deca5b6f9cbd971790debf5d890e560ef56d52597679275771209c898bc1"
Oct 30 19:17:14.923991 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.923907    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2164deca5b6f9cbd971790debf5d890e560ef56d52597679275771209c898bc1"} err="failed to get container status \"2164deca5b6f9cbd971790debf5d890e560ef56d52597679275771209c898bc1\": rpc error: code = NotFound desc = could not find container \"2164deca5b6f9cbd971790debf5d890e560ef56d52597679275771209c898bc1\": container with ID starting with 2164deca5b6f9cbd971790debf5d890e560ef56d52597679275771209c898bc1 not found: ID does not exist"
Oct 30 19:17:14.944958 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.944928    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j84j7\" (UniqueName: \"kubernetes.io/projected/dc3baffc-49c4-4101-9bf4-047229d89000-kube-api-access-j84j7\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:14.997023 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:14.996986    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"]
Oct 30 19:17:15.002499 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:15.002470    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a1617e-openshift-default-75c6f6977c-hkq7w"]
Oct 30 19:17:15.014182 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:15.014163    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41e9e11-9wrkt"
Oct 30 19:17:15.045439 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:15.045411    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/996da56a-ede7-4041-b0c4-1d5f10f1b659-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/996da56a-ede7-4041-b0c4-1d5f10f1b659-maas-api-openshift-ca\") pod \"996da56a-ede7-4041-b0c4-1d5f10f1b659\" (UID: \"996da56a-ede7-4041-b0c4-1d5f10f1b659\") "
Oct 30 19:17:15.045569 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:15.045462    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/996da56a-ede7-4041-b0c4-1d5f10f1b659-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/996da56a-ede7-4041-b0c4-1d5f10f1b659-maas-api-xks-ca\") pod \"996da56a-ede7-4041-b0c4-1d5f10f1b659\" (UID: \"996da56a-ede7-4041-b0c4-1d5f10f1b659\") "
Oct 30 19:17:15.045569 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:15.045546    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/996da56a-ede7-4041-b0c4-1d5f10f1b659-kube-api-access-xzt4m\" (UniqueName: \"kubernetes.io/projected/996da56a-ede7-4041-b0c4-1d5f10f1b659-kube-api-access-xzt4m\") pod \"996da56a-ede7-4041-b0c4-1d5f10f1b659\" (UID: \"996da56a-ede7-4041-b0c4-1d5f10f1b659\") "
Oct 30 19:17:15.045747 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:15.045726    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/996da56a-ede7-4041-b0c4-1d5f10f1b659-maas-api-openshift-ca" pod "996da56a-ede7-4041-b0c4-1d5f10f1b659" (UID: "996da56a-ede7-4041-b0c4-1d5f10f1b659"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:15.045993 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:15.045964    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/996da56a-ede7-4041-b0c4-1d5f10f1b659-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:15.047448 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:15.047428    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/996da56a-ede7-4041-b0c4-1d5f10f1b659-maas-api-xks-ca" pod "996da56a-ede7-4041-b0c4-1d5f10f1b659" (UID: "996da56a-ede7-4041-b0c4-1d5f10f1b659"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:15.047611 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:15.047578    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996da56a-ede7-4041-b0c4-1d5f10f1b659-kube-api-access-xzt4m" pod "996da56a-ede7-4041-b0c4-1d5f10f1b659" (UID: "996da56a-ede7-4041-b0c4-1d5f10f1b659"). InnerVolumeSpecName "kube-api-access-xzt4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:15.147555 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:15.147524    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzt4m\" (UniqueName: \"kubernetes.io/projected/996da56a-ede7-4041-b0c4-1d5f10f1b659-kube-api-access-xzt4m\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:15.147555 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:15.147551    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/996da56a-ede7-4041-b0c4-1d5f10f1b659-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:15.168669 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:15.168596    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d"]
Oct 30 19:17:15.174714 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:15.174687    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ec8dab-7bd4cd476-h892d"]
Oct 30 19:17:15.339147 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:15.339116    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf"]
Oct 30 19:17:15.344175 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:15.344147    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ec8dab-6dd5c58bff-9pnkf"]
Oct 30 19:17:15.711410 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:17:15.711375    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:17:15.891642 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:15.891599    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41e9e11-9wrkt" event={"ID":"996da56a-ede7-4041-b0c4-1d5f10f1b659","Type":"ContainerDied","Data":"d271d045ba75f4fff3eef5be9e263153ce2d322b86eb9fc7ba97ee9b8d0122fb"}
Oct 30 19:17:15.892150 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:15.891672    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d271d045ba75f4fff3eef5be9e263153ce2d322b86eb9fc7ba97ee9b8d0122fb"
Oct 30 19:17:15.892150 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:15.891613    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41e9e11-9wrkt"
Oct 30 19:17:16.870134 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:16.866708    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9cd1a58-ecd4-4611-aaac-28109cd2981a" path="/var/lib/kubelet/pods/b9cd1a58-ecd4-4611-aaac-28109cd2981a/volumes"
Oct 30 19:17:16.870134 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:16.867461    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6ec64bd-1810-4f78-b451-9faae648be6e" path="/var/lib/kubelet/pods/d6ec64bd-1810-4f78-b451-9faae648be6e/volumes"
Oct 30 19:17:16.870134 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:16.867886    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc3baffc-49c4-4101-9bf4-047229d89000" path="/var/lib/kubelet/pods/dc3baffc-49c4-4101-9bf4-047229d89000/volumes"
Oct 30 19:17:16.882331 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:16.882300    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:21.729167 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:21.729129    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:22.225990 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.225945    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:22.513595 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.513519    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql"]
Oct 30 19:17:22.514114 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.514095    4924 cpu_manager.go:414] "containerMap: removing container" podUID="d6ec64bd-1810-4f78-b451-9faae648be6e" containerName="istio-proxy"
Oct 30 19:17:22.514173 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.514115    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6ec64bd-1810-4f78-b451-9faae648be6e" containerName="istio-proxy"
Oct 30 19:17:22.514173 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.514135    4924 cpu_manager.go:414] "containerMap: removing container" podUID="dc3baffc-49c4-4101-9bf4-047229d89000" containerName="payload-pre-processing"
Oct 30 19:17:22.514173 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.514141    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc3baffc-49c4-4101-9bf4-047229d89000" containerName="payload-pre-processing"
Oct 30 19:17:22.514173 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.514152    4924 cpu_manager.go:414] "containerMap: removing container" podUID="b9cd1a58-ecd4-4611-aaac-28109cd2981a" containerName="payload-processing"
Oct 30 19:17:22.514173 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.514157    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9cd1a58-ecd4-4611-aaac-28109cd2981a" containerName="payload-processing"
Oct 30 19:17:22.514173 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.514173    4924 cpu_manager.go:414] "containerMap: removing container" podUID="996da56a-ede7-4041-b0c4-1d5f10f1b659" containerName="revoke-keys"
Oct 30 19:17:22.514409 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.514179    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="996da56a-ede7-4041-b0c4-1d5f10f1b659" containerName="revoke-keys"
Oct 30 19:17:22.514409 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.514188    4924 cpu_manager.go:414] "containerMap: removing container" podUID="d6ec64bd-1810-4f78-b451-9faae648be6e" containerName="istio-proxy"
Oct 30 19:17:22.514409 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.514192    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6ec64bd-1810-4f78-b451-9faae648be6e" containerName="istio-proxy"
Oct 30 19:17:22.514409 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.514271    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9cd1a58-ecd4-4611-aaac-28109cd2981a" containerName="payload-processing"
Oct 30 19:17:22.514409 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.514285    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="996da56a-ede7-4041-b0c4-1d5f10f1b659" containerName="revoke-keys"
Oct 30 19:17:22.514409 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.514290    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6ec64bd-1810-4f78-b451-9faae648be6e" containerName="istio-proxy"
Oct 30 19:17:22.514409 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.514299    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6ec64bd-1810-4f78-b451-9faae648be6e" containerName="istio-proxy"
Oct 30 19:17:22.514409 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.514306    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc3baffc-49c4-4101-9bf4-047229d89000" containerName="payload-pre-processing"
Oct 30 19:17:22.518864 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.518843    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql"
Oct 30 19:17:22.571359 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.571327    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql"]
Oct 30 19:17:22.614420 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.614392    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:17:22.614606 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.614588    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:17:22.614672 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.614609    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:17:22.614845 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.614828    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:17:22.633605 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.633581    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h9sg\" (UniqueName: \"kubernetes.io/projected/80c4bbbc-11d1-40a4-9def-e6d079a72a44-kube-api-access-9h9sg\") pod \"maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql\" (UID: \"80c4bbbc-11d1-40a4-9def-e6d079a72a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql"
Oct 30 19:17:22.633748 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.633622    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80c4bbbc-11d1-40a4-9def-e6d079a72a44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql\" (UID: \"80c4bbbc-11d1-40a4-9def-e6d079a72a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql"
Oct 30 19:17:22.633792 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.633769    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80c4bbbc-11d1-40a4-9def-e6d079a72a44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql\" (UID: \"80c4bbbc-11d1-40a4-9def-e6d079a72a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql"
Oct 30 19:17:22.734686 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.734660    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h9sg\" (UniqueName: \"kubernetes.io/projected/80c4bbbc-11d1-40a4-9def-e6d079a72a44-kube-api-access-9h9sg\") pod \"maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql\" (UID: \"80c4bbbc-11d1-40a4-9def-e6d079a72a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql"
Oct 30 19:17:22.735094 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.734699    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80c4bbbc-11d1-40a4-9def-e6d079a72a44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql\" (UID: \"80c4bbbc-11d1-40a4-9def-e6d079a72a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql"
Oct 30 19:17:22.735094 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.734770    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80c4bbbc-11d1-40a4-9def-e6d079a72a44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql\" (UID: \"80c4bbbc-11d1-40a4-9def-e6d079a72a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql"
Oct 30 19:17:22.735518 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.735498    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80c4bbbc-11d1-40a4-9def-e6d079a72a44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql\" (UID: \"80c4bbbc-11d1-40a4-9def-e6d079a72a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql"
Oct 30 19:17:22.737194 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.737177    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80c4bbbc-11d1-40a4-9def-e6d079a72a44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql\" (UID: \"80c4bbbc-11d1-40a4-9def-e6d079a72a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql"
Oct 30 19:17:22.789398 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.789342    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h9sg\" (UniqueName: \"kubernetes.io/projected/80c4bbbc-11d1-40a4-9def-e6d079a72a44-kube-api-access-9h9sg\") pod \"maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql\" (UID: \"80c4bbbc-11d1-40a4-9def-e6d079a72a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql"
Oct 30 19:17:22.833853 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:22.833831    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql"
Oct 30 19:17:23.201477 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:23.201449    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql"]
Oct 30 19:17:23.203786 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:17:23.203754    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c4bbbc_11d1_40a4_9def_e6d079a72a44.slice/crio-4c50851e58ef61177e6f3ef1e1823e19f39f0268c4caf3104341fdb7f6ceea85 WatchSource:0}: Error finding container 4c50851e58ef61177e6f3ef1e1823e19f39f0268c4caf3104341fdb7f6ceea85: Status 404 returned error can't find the container with id 4c50851e58ef61177e6f3ef1e1823e19f39f0268c4caf3104341fdb7f6ceea85
Oct 30 19:17:24.018498 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:24.018470    4924 generic.go:356] "Generic (PLEG): container finished" podID="80c4bbbc-11d1-40a4-9def-e6d079a72a44" containerID="a13d07abbe8b43479c9269b4b14c8f96d0129c626e4a664f63d2522a7c873dd5" exitCode=0
Oct 30 19:17:24.018890 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:24.018536    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql" event={"ID":"80c4bbbc-11d1-40a4-9def-e6d079a72a44","Type":"ContainerDied","Data":"a13d07abbe8b43479c9269b4b14c8f96d0129c626e4a664f63d2522a7c873dd5"}
Oct 30 19:17:24.018890 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:24.018571    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql" event={"ID":"80c4bbbc-11d1-40a4-9def-e6d079a72a44","Type":"ContainerStarted","Data":"4c50851e58ef61177e6f3ef1e1823e19f39f0268c4caf3104341fdb7f6ceea85"}
Oct 30 19:17:25.167193 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:25.167165    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql"
Oct 30 19:17:25.265570 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:25.265538    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80c4bbbc-11d1-40a4-9def-e6d079a72a44-kube-api-access-9h9sg\" (UniqueName: \"kubernetes.io/projected/80c4bbbc-11d1-40a4-9def-e6d079a72a44-kube-api-access-9h9sg\") pod \"80c4bbbc-11d1-40a4-9def-e6d079a72a44\" (UID: \"80c4bbbc-11d1-40a4-9def-e6d079a72a44\") "
Oct 30 19:17:25.265744 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:25.265617    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80c4bbbc-11d1-40a4-9def-e6d079a72a44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80c4bbbc-11d1-40a4-9def-e6d079a72a44-maas-api-openshift-ca\") pod \"80c4bbbc-11d1-40a4-9def-e6d079a72a44\" (UID: \"80c4bbbc-11d1-40a4-9def-e6d079a72a44\") "
Oct 30 19:17:25.265744 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:25.265715    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80c4bbbc-11d1-40a4-9def-e6d079a72a44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80c4bbbc-11d1-40a4-9def-e6d079a72a44-maas-api-xks-ca\") pod \"80c4bbbc-11d1-40a4-9def-e6d079a72a44\" (UID: \"80c4bbbc-11d1-40a4-9def-e6d079a72a44\") "
Oct 30 19:17:25.265968 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:25.265944    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c4bbbc-11d1-40a4-9def-e6d079a72a44-maas-api-openshift-ca" pod "80c4bbbc-11d1-40a4-9def-e6d079a72a44" (UID: "80c4bbbc-11d1-40a4-9def-e6d079a72a44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:25.266139 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:25.266121    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80c4bbbc-11d1-40a4-9def-e6d079a72a44-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:25.267706 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:25.267680    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80c4bbbc-11d1-40a4-9def-e6d079a72a44-maas-api-xks-ca" pod "80c4bbbc-11d1-40a4-9def-e6d079a72a44" (UID: "80c4bbbc-11d1-40a4-9def-e6d079a72a44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:25.267800 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:25.267703    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c4bbbc-11d1-40a4-9def-e6d079a72a44-kube-api-access-9h9sg" pod "80c4bbbc-11d1-40a4-9def-e6d079a72a44" (UID: "80c4bbbc-11d1-40a4-9def-e6d079a72a44"). InnerVolumeSpecName "kube-api-access-9h9sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:25.366956 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:25.366924    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80c4bbbc-11d1-40a4-9def-e6d079a72a44-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:25.366956 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:25.366955    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h9sg\" (UniqueName: \"kubernetes.io/projected/80c4bbbc-11d1-40a4-9def-e6d079a72a44-kube-api-access-9h9sg\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:26.035317 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:26.035288    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql"
Oct 30 19:17:26.035529 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:26.035281    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql" event={"ID":"80c4bbbc-11d1-40a4-9def-e6d079a72a44","Type":"ContainerDied","Data":"4c50851e58ef61177e6f3ef1e1823e19f39f0268c4caf3104341fdb7f6ceea85"}
Oct 30 19:17:26.035529 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:26.035412    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c50851e58ef61177e6f3ef1e1823e19f39f0268c4caf3104341fdb7f6ceea85"
Oct 30 19:17:26.100341 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:26.100306    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:27.191971 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:27.191938    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"]
Oct 30 19:17:27.238721 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:27.238691    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"]
Oct 30 19:17:27.345754 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:27.345724    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:27.675789 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:27.675755    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:27.997306 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:27.997223    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:28.223096 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:28.221925    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9"]
Oct 30 19:17:28.263148 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:28.263030    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-75e91b-9dfr9"]
Oct 30 19:17:28.671528 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:28.671497    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:28.834834 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:28.834800    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38a245bb-3de4-43a8-a017-e390e1092309" path="/var/lib/kubelet/pods/38a245bb-3de4-43a8-a017-e390e1092309/volumes"
Oct 30 19:17:28.994883 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:28.994799    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:29.320998 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:29.320899    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:29.540460 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:29.540430    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p"]
Oct 30 19:17:29.587167 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:29.587088    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec8dab-gkk6p"]
Oct 30 19:17:30.045317 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:30.045278    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:30.834561 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:30.834525    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7a23dbd-97df-45af-b79b-87ead5172a61" path="/var/lib/kubelet/pods/f7a23dbd-97df-45af-b79b-87ead5172a61/volumes"
Oct 30 19:17:32.821450 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:32.821410    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:33.813181 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:33.813148    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8260ef6-n25cb"]
Oct 30 19:17:33.813610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:33.813596    4924 cpu_manager.go:414] "containerMap: removing container" podUID="80c4bbbc-11d1-40a4-9def-e6d079a72a44" containerName="revoke-keys"
Oct 30 19:17:33.813660 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:33.813611    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80c4bbbc-11d1-40a4-9def-e6d079a72a44" containerName="revoke-keys"
Oct 30 19:17:33.813702 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:33.813698    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="80c4bbbc-11d1-40a4-9def-e6d079a72a44" containerName="revoke-keys"
Oct 30 19:17:33.818366 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:33.818345    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8260ef6-n25cb"
Oct 30 19:17:33.871728 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:33.871684    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddb6c82f-g4889"]
Oct 30 19:17:33.875833 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:33.875806    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb6c82f-g4889"
Oct 30 19:17:33.915318 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:33.915288    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:17:33.915496 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:33.915288    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:17:33.915496 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:33.915323    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:17:33.918221 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:33.918196    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:17:33.921206 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:33.921185    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8260ef6-n25cb"]
Oct 30 19:17:33.929903 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:33.929878    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddb6c82f-g4889"]
Oct 30 19:17:33.971270 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:33.971240    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03af0d78-56e2-44ab-9e4f-32f8aa9363a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddb6c82f-g4889\" (UID: \"03af0d78-56e2-44ab-9e4f-32f8aa9363a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb6c82f-g4889"
Oct 30 19:17:33.971402 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:33.971381    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98c2b6d-a534-43ef-bee4-780bf3f30539-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8260ef6-n25cb\" (UID: \"f98c2b6d-a534-43ef-bee4-780bf3f30539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8260ef6-n25cb"
Oct 30 19:17:33.971446 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:33.971406    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zqtx\" (UniqueName: \"kubernetes.io/projected/f98c2b6d-a534-43ef-bee4-780bf3f30539-kube-api-access-6zqtx\") pod \"maas-api-revoke-sub-f8260ef6-n25cb\" (UID: \"f98c2b6d-a534-43ef-bee4-780bf3f30539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8260ef6-n25cb"
Oct 30 19:17:33.971485 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:33.971461    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98c2b6d-a534-43ef-bee4-780bf3f30539-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8260ef6-n25cb\" (UID: \"f98c2b6d-a534-43ef-bee4-780bf3f30539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8260ef6-n25cb"
Oct 30 19:17:33.971577 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:33.971558    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03af0d78-56e2-44ab-9e4f-32f8aa9363a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddb6c82f-g4889\" (UID: \"03af0d78-56e2-44ab-9e4f-32f8aa9363a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb6c82f-g4889"
Oct 30 19:17:33.971614 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:33.971586    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdxb5\" (UniqueName: \"kubernetes.io/projected/03af0d78-56e2-44ab-9e4f-32f8aa9363a8-kube-api-access-gdxb5\") pod \"maas-api-revoke-sub-ddb6c82f-g4889\" (UID: \"03af0d78-56e2-44ab-9e4f-32f8aa9363a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb6c82f-g4889"
Oct 30 19:17:33.988418 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:33.988393    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq"]
Oct 30 19:17:34.072668 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:34.072592    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98c2b6d-a534-43ef-bee4-780bf3f30539-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8260ef6-n25cb\" (UID: \"f98c2b6d-a534-43ef-bee4-780bf3f30539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8260ef6-n25cb"
Oct 30 19:17:34.072668 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:34.072630    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zqtx\" (UniqueName: \"kubernetes.io/projected/f98c2b6d-a534-43ef-bee4-780bf3f30539-kube-api-access-6zqtx\") pod \"maas-api-revoke-sub-f8260ef6-n25cb\" (UID: \"f98c2b6d-a534-43ef-bee4-780bf3f30539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8260ef6-n25cb"
Oct 30 19:17:34.072887 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:34.072812    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98c2b6d-a534-43ef-bee4-780bf3f30539-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8260ef6-n25cb\" (UID: \"f98c2b6d-a534-43ef-bee4-780bf3f30539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8260ef6-n25cb"
Oct 30 19:17:34.072887 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:34.072872    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03af0d78-56e2-44ab-9e4f-32f8aa9363a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddb6c82f-g4889\" (UID: \"03af0d78-56e2-44ab-9e4f-32f8aa9363a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb6c82f-g4889"
Oct 30 19:17:34.073001 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:34.072892    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdxb5\" (UniqueName: \"kubernetes.io/projected/03af0d78-56e2-44ab-9e4f-32f8aa9363a8-kube-api-access-gdxb5\") pod \"maas-api-revoke-sub-ddb6c82f-g4889\" (UID: \"03af0d78-56e2-44ab-9e4f-32f8aa9363a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb6c82f-g4889"
Oct 30 19:17:34.073001 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:34.072925    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03af0d78-56e2-44ab-9e4f-32f8aa9363a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddb6c82f-g4889\" (UID: \"03af0d78-56e2-44ab-9e4f-32f8aa9363a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb6c82f-g4889"
Oct 30 19:17:34.073661 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:34.073632    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98c2b6d-a534-43ef-bee4-780bf3f30539-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8260ef6-n25cb\" (UID: \"f98c2b6d-a534-43ef-bee4-780bf3f30539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8260ef6-n25cb"
Oct 30 19:17:34.073796 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:34.073765    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03af0d78-56e2-44ab-9e4f-32f8aa9363a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddb6c82f-g4889\" (UID: \"03af0d78-56e2-44ab-9e4f-32f8aa9363a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb6c82f-g4889"
Oct 30 19:17:34.075166 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:34.075148    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98c2b6d-a534-43ef-bee4-780bf3f30539-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8260ef6-n25cb\" (UID: \"f98c2b6d-a534-43ef-bee4-780bf3f30539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8260ef6-n25cb"
Oct 30 19:17:34.075355 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:34.075333    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03af0d78-56e2-44ab-9e4f-32f8aa9363a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddb6c82f-g4889\" (UID: \"03af0d78-56e2-44ab-9e4f-32f8aa9363a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb6c82f-g4889"
Oct 30 19:17:34.127278 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:34.127248    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zqtx\" (UniqueName: \"kubernetes.io/projected/f98c2b6d-a534-43ef-bee4-780bf3f30539-kube-api-access-6zqtx\") pod \"maas-api-revoke-sub-f8260ef6-n25cb\" (UID: \"f98c2b6d-a534-43ef-bee4-780bf3f30539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8260ef6-n25cb"
Oct 30 19:17:34.127516 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:34.127493    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdxb5\" (UniqueName: \"kubernetes.io/projected/03af0d78-56e2-44ab-9e4f-32f8aa9363a8-kube-api-access-gdxb5\") pod \"maas-api-revoke-sub-ddb6c82f-g4889\" (UID: \"03af0d78-56e2-44ab-9e4f-32f8aa9363a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb6c82f-g4889"
Oct 30 19:17:34.131897 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:34.131877    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8260ef6-n25cb"
Oct 30 19:17:34.191498 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:34.190443    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb6c82f-g4889"
Oct 30 19:17:34.222479 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:34.222444    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:34.520653 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:17:34.520473    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98c2b6d_a534_43ef_bee4_780bf3f30539.slice/crio-3b97fa06b5022cb1f17736e03bc9d77d5e44a8eb7bb69c26ebae950083934f52 WatchSource:0}: Error finding container 3b97fa06b5022cb1f17736e03bc9d77d5e44a8eb7bb69c26ebae950083934f52: Status 404 returned error can't find the container with id 3b97fa06b5022cb1f17736e03bc9d77d5e44a8eb7bb69c26ebae950083934f52
Oct 30 19:17:34.521393 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:34.521360    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8260ef6-n25cb"]
Oct 30 19:17:34.554256 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:34.554227    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:34.612397 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:34.612365    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddb6c82f-g4889"]
Oct 30 19:17:34.615307 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:17:34.615275    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03af0d78_56e2_44ab_9e4f_32f8aa9363a8.slice/crio-cddf19c764cf03deda68697a2b5c97facb3853293d7aa5d38b417215f8c1091d WatchSource:0}: Error finding container cddf19c764cf03deda68697a2b5c97facb3853293d7aa5d38b417215f8c1091d: Status 404 returned error can't find the container with id cddf19c764cf03deda68697a2b5c97facb3853293d7aa5d38b417215f8c1091d
Oct 30 19:17:34.872358 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:34.872327    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:35.100573 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:35.100544    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb6c82f-g4889" event={"ID":"03af0d78-56e2-44ab-9e4f-32f8aa9363a8","Type":"ContainerStarted","Data":"e12c6d9e72f9fac561cd81264c32689580bb93745f7b279980c23471eea4ecfb"}
Oct 30 19:17:35.100753 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:35.100597    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb6c82f-g4889" event={"ID":"03af0d78-56e2-44ab-9e4f-32f8aa9363a8","Type":"ContainerStarted","Data":"cddf19c764cf03deda68697a2b5c97facb3853293d7aa5d38b417215f8c1091d"}
Oct 30 19:17:35.102229 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:35.102202    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8260ef6-n25cb" event={"ID":"f98c2b6d-a534-43ef-bee4-780bf3f30539","Type":"ContainerStarted","Data":"b44dd6454401bde5aa192026ec3f37ce319a8cda03bd826a19602135ed60a73d"}
Oct 30 19:17:35.102362 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:35.102244    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8260ef6-n25cb" event={"ID":"f98c2b6d-a534-43ef-bee4-780bf3f30539","Type":"ContainerStarted","Data":"3b97fa06b5022cb1f17736e03bc9d77d5e44a8eb7bb69c26ebae950083934f52"}
Oct 30 19:17:35.212406 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:35.212352    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb6c82f-g4889" podStartSLOduration=2.212338135 podStartE2EDuration="2.212338135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:35.210994942 +0000 UTC m=+2456.488045388" watchObservedRunningTime="2026-10-30 19:17:35.212338135 +0000 UTC m=+2456.489388580"
Oct 30 19:17:35.321374 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:35.321319    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8260ef6-n25cb" podStartSLOduration=2.321304986 podStartE2EDuration="2.321304986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:35.319725207 +0000 UTC m=+2456.596775651" watchObservedRunningTime="2026-10-30 19:17:35.321304986 +0000 UTC m=+2456.598355431"
Oct 30 19:17:35.975923 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:35.975893    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:36.111260 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:36.111229    4924 generic.go:356] "Generic (PLEG): container finished" podID="03af0d78-56e2-44ab-9e4f-32f8aa9363a8" containerID="e12c6d9e72f9fac561cd81264c32689580bb93745f7b279980c23471eea4ecfb" exitCode=0
Oct 30 19:17:36.111471 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:36.111322    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb6c82f-g4889" event={"ID":"03af0d78-56e2-44ab-9e4f-32f8aa9363a8","Type":"ContainerDied","Data":"e12c6d9e72f9fac561cd81264c32689580bb93745f7b279980c23471eea4ecfb"}
Oct 30 19:17:36.113108 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:36.113052    4924 generic.go:356] "Generic (PLEG): container finished" podID="f98c2b6d-a534-43ef-bee4-780bf3f30539" containerID="b44dd6454401bde5aa192026ec3f37ce319a8cda03bd826a19602135ed60a73d" exitCode=0
Oct 30 19:17:36.113108 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:36.113093    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8260ef6-n25cb" event={"ID":"f98c2b6d-a534-43ef-bee4-780bf3f30539","Type":"ContainerDied","Data":"b44dd6454401bde5aa192026ec3f37ce319a8cda03bd826a19602135ed60a73d"}
Oct 30 19:17:36.274427 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:36.274344    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"]
Oct 30 19:17:36.274604 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:36.274545    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv" podUID="4dab0473-fa74-460f-8b18-69874157c8fa" containerName="istio-proxy" containerID="cri-o://8d2c6380a2d90f3b4be291705554417d2ef61505b1c901eb7534f3058e715aef" gracePeriod=30
Oct 30 19:17:36.394325 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:36.394155    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:37.035998 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:37.035961    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:37.292831 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:37.292774    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb6c82f-g4889"
Oct 30 19:17:37.295533 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:37.295513    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8260ef6-n25cb"
Oct 30 19:17:37.334844 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:37.334821    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03af0d78-56e2-44ab-9e4f-32f8aa9363a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03af0d78-56e2-44ab-9e4f-32f8aa9363a8-maas-api-openshift-ca\") pod \"03af0d78-56e2-44ab-9e4f-32f8aa9363a8\" (UID: \"03af0d78-56e2-44ab-9e4f-32f8aa9363a8\") "
Oct 30 19:17:37.334972 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:37.334936    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03af0d78-56e2-44ab-9e4f-32f8aa9363a8-kube-api-access-gdxb5\" (UniqueName: \"kubernetes.io/projected/03af0d78-56e2-44ab-9e4f-32f8aa9363a8-kube-api-access-gdxb5\") pod \"03af0d78-56e2-44ab-9e4f-32f8aa9363a8\" (UID: \"03af0d78-56e2-44ab-9e4f-32f8aa9363a8\") "
Oct 30 19:17:37.335043 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:37.334990    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03af0d78-56e2-44ab-9e4f-32f8aa9363a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03af0d78-56e2-44ab-9e4f-32f8aa9363a8-maas-api-xks-ca\") pod \"03af0d78-56e2-44ab-9e4f-32f8aa9363a8\" (UID: \"03af0d78-56e2-44ab-9e4f-32f8aa9363a8\") "
Oct 30 19:17:37.335122 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:37.335036    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f98c2b6d-a534-43ef-bee4-780bf3f30539-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98c2b6d-a534-43ef-bee4-780bf3f30539-maas-api-openshift-ca\") pod \"f98c2b6d-a534-43ef-bee4-780bf3f30539\" (UID: \"f98c2b6d-a534-43ef-bee4-780bf3f30539\") "
Oct 30 19:17:37.335187 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:37.335154    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f98c2b6d-a534-43ef-bee4-780bf3f30539-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98c2b6d-a534-43ef-bee4-780bf3f30539-maas-api-xks-ca\") pod \"f98c2b6d-a534-43ef-bee4-780bf3f30539\" (UID: \"f98c2b6d-a534-43ef-bee4-780bf3f30539\") "
Oct 30 19:17:37.335244 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:37.335184    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f98c2b6d-a534-43ef-bee4-780bf3f30539-kube-api-access-6zqtx\" (UniqueName: \"kubernetes.io/projected/f98c2b6d-a534-43ef-bee4-780bf3f30539-kube-api-access-6zqtx\") pod \"f98c2b6d-a534-43ef-bee4-780bf3f30539\" (UID: \"f98c2b6d-a534-43ef-bee4-780bf3f30539\") "
Oct 30 19:17:37.335244 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:37.335203    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03af0d78-56e2-44ab-9e4f-32f8aa9363a8-maas-api-openshift-ca" pod "03af0d78-56e2-44ab-9e4f-32f8aa9363a8" (UID: "03af0d78-56e2-44ab-9e4f-32f8aa9363a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:37.335510 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:37.335482    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98c2b6d-a534-43ef-bee4-780bf3f30539-maas-api-openshift-ca" pod "f98c2b6d-a534-43ef-bee4-780bf3f30539" (UID: "f98c2b6d-a534-43ef-bee4-780bf3f30539"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:37.335707 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:37.335687    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98c2b6d-a534-43ef-bee4-780bf3f30539-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:37.335782 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:37.335709    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03af0d78-56e2-44ab-9e4f-32f8aa9363a8-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:37.337298 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:37.337273    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f98c2b6d-a534-43ef-bee4-780bf3f30539-maas-api-xks-ca" pod "f98c2b6d-a534-43ef-bee4-780bf3f30539" (UID: "f98c2b6d-a534-43ef-bee4-780bf3f30539"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:37.337298 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:37.337282    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98c2b6d-a534-43ef-bee4-780bf3f30539-kube-api-access-6zqtx" pod "f98c2b6d-a534-43ef-bee4-780bf3f30539" (UID: "f98c2b6d-a534-43ef-bee4-780bf3f30539"). InnerVolumeSpecName "kube-api-access-6zqtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:37.337535 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:37.337514    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03af0d78-56e2-44ab-9e4f-32f8aa9363a8-kube-api-access-gdxb5" pod "03af0d78-56e2-44ab-9e4f-32f8aa9363a8" (UID: "03af0d78-56e2-44ab-9e4f-32f8aa9363a8"). InnerVolumeSpecName "kube-api-access-gdxb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:37.337535 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:37.337521    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03af0d78-56e2-44ab-9e4f-32f8aa9363a8-maas-api-xks-ca" pod "03af0d78-56e2-44ab-9e4f-32f8aa9363a8" (UID: "03af0d78-56e2-44ab-9e4f-32f8aa9363a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:37.437102 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:37.437042    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98c2b6d-a534-43ef-bee4-780bf3f30539-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:37.437102 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:37.437103    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zqtx\" (UniqueName: \"kubernetes.io/projected/f98c2b6d-a534-43ef-bee4-780bf3f30539-kube-api-access-6zqtx\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:37.437316 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:37.437120    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdxb5\" (UniqueName: \"kubernetes.io/projected/03af0d78-56e2-44ab-9e4f-32f8aa9363a8-kube-api-access-gdxb5\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:37.437316 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:37.437134    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03af0d78-56e2-44ab-9e4f-32f8aa9363a8-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:37.442917 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:37.442871    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"]
Oct 30 19:17:37.443177 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:37.443146    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz" podUID="467dd97e-420c-44ac-a249-17a3deacfb63" containerName="istio-proxy" containerID="cri-o://de74ea692d63df6c2372d51537836a2f9171a9a0890b72e08b0506c6818fd512" gracePeriod=30
Oct 30 19:17:37.492413 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:37.492379    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:37.844397 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:37.844346    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:38.128553 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:38.128526    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb6c82f-g4889"
Oct 30 19:17:38.128969 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:38.128558    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb6c82f-g4889" event={"ID":"03af0d78-56e2-44ab-9e4f-32f8aa9363a8","Type":"ContainerDied","Data":"cddf19c764cf03deda68697a2b5c97facb3853293d7aa5d38b417215f8c1091d"}
Oct 30 19:17:38.128969 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:38.128609    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cddf19c764cf03deda68697a2b5c97facb3853293d7aa5d38b417215f8c1091d"
Oct 30 19:17:38.130091 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:38.130053    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8260ef6-n25cb"
Oct 30 19:17:38.130215 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:38.130087    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8260ef6-n25cb" event={"ID":"f98c2b6d-a534-43ef-bee4-780bf3f30539","Type":"ContainerDied","Data":"3b97fa06b5022cb1f17736e03bc9d77d5e44a8eb7bb69c26ebae950083934f52"}
Oct 30 19:17:38.130215 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:38.130140    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b97fa06b5022cb1f17736e03bc9d77d5e44a8eb7bb69c26ebae950083934f52"
Oct 30 19:17:38.677249 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:38.677219    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:38.989290 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:38.989205    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq" podUID="539e8daa-2f24-4d43-8017-237fb777d90e" containerName="payload-pre-processing" containerID="cri-o://d8cbacada5c40930e6550b79d44fc1752c4ec5f2df7884738014c20e0cc0eb04" gracePeriod=25
Oct 30 19:17:39.466661 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:39.466620    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq" podUID="539e8daa-2f24-4d43-8017-237fb777d90e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:17:39.599850 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:39.599816    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:40.778826 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.778794    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"]
Oct 30 19:17:40.779324 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.779306    4924 cpu_manager.go:414] "containerMap: removing container" podUID="03af0d78-56e2-44ab-9e4f-32f8aa9363a8" containerName="revoke-keys"
Oct 30 19:17:40.779366 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.779324    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03af0d78-56e2-44ab-9e4f-32f8aa9363a8" containerName="revoke-keys"
Oct 30 19:17:40.779366 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.779350    4924 cpu_manager.go:414] "containerMap: removing container" podUID="f98c2b6d-a534-43ef-bee4-780bf3f30539" containerName="revoke-keys"
Oct 30 19:17:40.779366 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.779355    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f98c2b6d-a534-43ef-bee4-780bf3f30539" containerName="revoke-keys"
Oct 30 19:17:40.779465 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.779436    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="03af0d78-56e2-44ab-9e4f-32f8aa9363a8" containerName="revoke-keys"
Oct 30 19:17:40.779465 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.779450    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="f98c2b6d-a534-43ef-bee4-780bf3f30539" containerName="revoke-keys"
Oct 30 19:17:40.782663 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.782642    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"
Oct 30 19:17:40.873122 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.873087    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcdnw\" (UniqueName: \"kubernetes.io/projected/9bbf86bc-d98c-4c76-b96e-ff9a48795800-kube-api-access-bcdnw\") pod \"test-model-f75d5f-kserve-7767fbbdf-4fmk5\" (UID: \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\") " pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"
Oct 30 19:17:40.873297 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.873136    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-home\") pod \"test-model-f75d5f-kserve-7767fbbdf-4fmk5\" (UID: \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\") " pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"
Oct 30 19:17:40.873347 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.873296    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-model-cache\") pod \"test-model-f75d5f-kserve-7767fbbdf-4fmk5\" (UID: \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\") " pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"
Oct 30 19:17:40.873347 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.873339    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bbf86bc-d98c-4c76-b96e-ff9a48795800-tls-certs\") pod \"test-model-f75d5f-kserve-7767fbbdf-4fmk5\" (UID: \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\") " pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"
Oct 30 19:17:40.873448 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.873435    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-tmp-dir\") pod \"test-model-f75d5f-kserve-7767fbbdf-4fmk5\" (UID: \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\") " pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"
Oct 30 19:17:40.873498 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.873464    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-dshm\") pod \"test-model-f75d5f-kserve-7767fbbdf-4fmk5\" (UID: \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\") " pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"
Oct 30 19:17:40.878816 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.878794    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f75d5f\"/\"openshift-service-ca.crt\""
Oct 30 19:17:40.878980 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.878830    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f75d5f\"/\"test-model-f75d5f-kserve-self-signed-certs\""
Oct 30 19:17:40.878980 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.878904    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f75d5f\"/\"kube-root-ca.crt\""
Oct 30 19:17:40.879109 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.878983    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f75d5f\"/\"default-dockercfg-h6fw5\""
Oct 30 19:17:40.882414 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.882396    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"]
Oct 30 19:17:40.886321 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.886296    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:40.974461 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.974427    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-dshm\") pod \"test-model-f75d5f-kserve-7767fbbdf-4fmk5\" (UID: \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\") " pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"
Oct 30 19:17:40.974624 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.974539    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcdnw\" (UniqueName: \"kubernetes.io/projected/9bbf86bc-d98c-4c76-b96e-ff9a48795800-kube-api-access-bcdnw\") pod \"test-model-f75d5f-kserve-7767fbbdf-4fmk5\" (UID: \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\") " pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"
Oct 30 19:17:40.974624 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.974575    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-home\") pod \"test-model-f75d5f-kserve-7767fbbdf-4fmk5\" (UID: \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\") " pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"
Oct 30 19:17:40.974624 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.974615    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-model-cache\") pod \"test-model-f75d5f-kserve-7767fbbdf-4fmk5\" (UID: \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\") " pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"
Oct 30 19:17:40.974774 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.974637    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bbf86bc-d98c-4c76-b96e-ff9a48795800-tls-certs\") pod \"test-model-f75d5f-kserve-7767fbbdf-4fmk5\" (UID: \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\") " pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"
Oct 30 19:17:40.974774 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.974668    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-tmp-dir\") pod \"test-model-f75d5f-kserve-7767fbbdf-4fmk5\" (UID: \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\") " pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"
Oct 30 19:17:40.974968 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.974944    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-home\") pod \"test-model-f75d5f-kserve-7767fbbdf-4fmk5\" (UID: \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\") " pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"
Oct 30 19:17:40.975057 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.974988    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-model-cache\") pod \"test-model-f75d5f-kserve-7767fbbdf-4fmk5\" (UID: \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\") " pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"
Oct 30 19:17:40.975057 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.975034    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-tmp-dir\") pod \"test-model-f75d5f-kserve-7767fbbdf-4fmk5\" (UID: \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\") " pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"
Oct 30 19:17:40.976850 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.976829    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-dshm\") pod \"test-model-f75d5f-kserve-7767fbbdf-4fmk5\" (UID: \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\") " pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"
Oct 30 19:17:40.977111 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:40.977094    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bbf86bc-d98c-4c76-b96e-ff9a48795800-tls-certs\") pod \"test-model-f75d5f-kserve-7767fbbdf-4fmk5\" (UID: \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\") " pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"
Oct 30 19:17:41.032237 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.032166    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcdnw\" (UniqueName: \"kubernetes.io/projected/9bbf86bc-d98c-4c76-b96e-ff9a48795800-kube-api-access-bcdnw\") pod \"test-model-f75d5f-kserve-7767fbbdf-4fmk5\" (UID: \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\") " pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"
Oct 30 19:17:41.077016 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.076983    4924 scope.go:122] "RemoveContainer" containerID="73373dfe5a4c261911414ffc16987c4260144075f27dc16229e4f2eff2c9301e"
Oct 30 19:17:41.087142 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.087122    4924 scope.go:122] "RemoveContainer" containerID="827a910a1ed81918a751825f88897fe449089ab857e688010a9c799607504fd8"
Oct 30 19:17:41.094263 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.094235    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"
Oct 30 19:17:41.213095 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.213050    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"]
Oct 30 19:17:41.218670 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.218651    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.279059 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.279021    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99520a11-7547-4a41-90e6-f03de8d3cccd-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.279272 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.279099    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99520a11-7547-4a41-90e6-f03de8d3cccd-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.279272 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.279197    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99520a11-7547-4a41-90e6-f03de8d3cccd-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.279402 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.279271    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99520a11-7547-4a41-90e6-f03de8d3cccd-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.279402 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.279329    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99520a11-7547-4a41-90e6-f03de8d3cccd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.279402 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.279363    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99520a11-7547-4a41-90e6-f03de8d3cccd-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.279618 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.279485    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktnwq\" (UniqueName: \"kubernetes.io/projected/99520a11-7547-4a41-90e6-f03de8d3cccd-kube-api-access-ktnwq\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.279618 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.279567    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99520a11-7547-4a41-90e6-f03de8d3cccd-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.279735 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.279646    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99520a11-7547-4a41-90e6-f03de8d3cccd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.282971 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.282897    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"]
Oct 30 19:17:41.381347 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.381300    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99520a11-7547-4a41-90e6-f03de8d3cccd-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.381347 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.381351    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99520a11-7547-4a41-90e6-f03de8d3cccd-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.381591 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.381393    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99520a11-7547-4a41-90e6-f03de8d3cccd-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.381591 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.381425    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99520a11-7547-4a41-90e6-f03de8d3cccd-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.381591 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.381457    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99520a11-7547-4a41-90e6-f03de8d3cccd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.381762 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.381628    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99520a11-7547-4a41-90e6-f03de8d3cccd-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.381762 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.381731    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktnwq\" (UniqueName: \"kubernetes.io/projected/99520a11-7547-4a41-90e6-f03de8d3cccd-kube-api-access-ktnwq\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.381762 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.381742    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99520a11-7547-4a41-90e6-f03de8d3cccd-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.381916 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.381766    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99520a11-7547-4a41-90e6-f03de8d3cccd-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.381916 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.381829    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99520a11-7547-4a41-90e6-f03de8d3cccd-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.381916 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.381888    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99520a11-7547-4a41-90e6-f03de8d3cccd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.382160 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.381989    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99520a11-7547-4a41-90e6-f03de8d3cccd-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.382238 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.382206    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99520a11-7547-4a41-90e6-f03de8d3cccd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.382291 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.382210    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99520a11-7547-4a41-90e6-f03de8d3cccd-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.383685 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.383663    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99520a11-7547-4a41-90e6-f03de8d3cccd-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.384209 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.384190    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99520a11-7547-4a41-90e6-f03de8d3cccd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.436519 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.436485    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99520a11-7547-4a41-90e6-f03de8d3cccd-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.436674 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.436600    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktnwq\" (UniqueName: \"kubernetes.io/projected/99520a11-7547-4a41-90e6-f03de8d3cccd-kube-api-access-ktnwq\") pod \"maas-default-gateway-openshift-default-75d7c785f-mh9pf\" (UID: \"99520a11-7547-4a41-90e6-f03de8d3cccd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.495999 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.493850    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"]
Oct 30 19:17:41.517877 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.517853    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 19:17:41.532900 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.532877    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:41.594528 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.594424    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-istio-data\") pod \"4dab0473-fa74-460f-8b18-69874157c8fa\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") "
Oct 30 19:17:41.594528 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.594474    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4dab0473-fa74-460f-8b18-69874157c8fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dab0473-fa74-460f-8b18-69874157c8fa-istio-podinfo\") pod \"4dab0473-fa74-460f-8b18-69874157c8fa\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") "
Oct 30 19:17:41.594528 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.594499    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dab0473-fa74-460f-8b18-69874157c8fa-istio-token\" (UniqueName: \"kubernetes.io/projected/4dab0473-fa74-460f-8b18-69874157c8fa-istio-token\") pod \"4dab0473-fa74-460f-8b18-69874157c8fa\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") "
Oct 30 19:17:41.594528 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.594555    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-istio-envoy\") pod \"4dab0473-fa74-460f-8b18-69874157c8fa\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") "
Oct 30 19:17:41.594528 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.594584    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-workload-certs\") pod \"4dab0473-fa74-460f-8b18-69874157c8fa\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") "
Oct 30 19:17:41.595482 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.594624    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dab0473-fa74-460f-8b18-69874157c8fa-kube-api-access-z9bjx\" (UniqueName: \"kubernetes.io/projected/4dab0473-fa74-460f-8b18-69874157c8fa-kube-api-access-z9bjx\") pod \"4dab0473-fa74-460f-8b18-69874157c8fa\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") "
Oct 30 19:17:41.595482 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.594664    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-credential-socket\") pod \"4dab0473-fa74-460f-8b18-69874157c8fa\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") "
Oct 30 19:17:41.595482 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.594691    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-workload-socket\") pod \"4dab0473-fa74-460f-8b18-69874157c8fa\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") "
Oct 30 19:17:41.595482 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.594766    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dab0473-fa74-460f-8b18-69874157c8fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dab0473-fa74-460f-8b18-69874157c8fa-istiod-ca-cert\") pod \"4dab0473-fa74-460f-8b18-69874157c8fa\" (UID: \"4dab0473-fa74-460f-8b18-69874157c8fa\") "
Oct 30 19:17:41.595482 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.594777    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-istio-data" pod "4dab0473-fa74-460f-8b18-69874157c8fa" (UID: "4dab0473-fa74-460f-8b18-69874157c8fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:41.595482 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.595058    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-credential-socket" pod "4dab0473-fa74-460f-8b18-69874157c8fa" (UID: "4dab0473-fa74-460f-8b18-69874157c8fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:41.595482 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.595341    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-workload-socket" pod "4dab0473-fa74-460f-8b18-69874157c8fa" (UID: "4dab0473-fa74-460f-8b18-69874157c8fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:41.595482 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.595370    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dab0473-fa74-460f-8b18-69874157c8fa-istiod-ca-cert" pod "4dab0473-fa74-460f-8b18-69874157c8fa" (UID: "4dab0473-fa74-460f-8b18-69874157c8fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:41.595482 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.595384    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-workload-certs" pod "4dab0473-fa74-460f-8b18-69874157c8fa" (UID: "4dab0473-fa74-460f-8b18-69874157c8fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:41.595482 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.595420    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:41.595482 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.595437    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:41.597538 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.597507    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dab0473-fa74-460f-8b18-69874157c8fa-istio-token" pod "4dab0473-fa74-460f-8b18-69874157c8fa" (UID: "4dab0473-fa74-460f-8b18-69874157c8fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:41.598837 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.598813    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-istio-envoy" pod "4dab0473-fa74-460f-8b18-69874157c8fa" (UID: "4dab0473-fa74-460f-8b18-69874157c8fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:41.598969 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.598819    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4dab0473-fa74-460f-8b18-69874157c8fa-istio-podinfo" pod "4dab0473-fa74-460f-8b18-69874157c8fa" (UID: "4dab0473-fa74-460f-8b18-69874157c8fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:41.600263 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.600215    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dab0473-fa74-460f-8b18-69874157c8fa-kube-api-access-z9bjx" pod "4dab0473-fa74-460f-8b18-69874157c8fa" (UID: "4dab0473-fa74-460f-8b18-69874157c8fa"). InnerVolumeSpecName "kube-api-access-z9bjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:41.696253 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.696220    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dab0473-fa74-460f-8b18-69874157c8fa-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:41.696253 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.696250    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dab0473-fa74-460f-8b18-69874157c8fa-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:41.696253 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.696259    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:41.696487 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.696268    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:41.696487 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.696277    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9bjx\" (UniqueName: \"kubernetes.io/projected/4dab0473-fa74-460f-8b18-69874157c8fa-kube-api-access-z9bjx\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:41.696487 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.696288    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dab0473-fa74-460f-8b18-69874157c8fa-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:41.696487 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.696296    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dab0473-fa74-460f-8b18-69874157c8fa-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:41.918045 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.917951    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"]
Oct 30 19:17:41.919334 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:17:41.919294    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99520a11_7547_4a41_90e6_f03de8d3cccd.slice/crio-6f57891bdc5c11721cb1a5b8fd9d093175a8f26c2d08d4c726f4b2907cf4863d WatchSource:0}: Error finding container 6f57891bdc5c11721cb1a5b8fd9d093175a8f26c2d08d4c726f4b2907cf4863d: Status 404 returned error can't find the container with id 6f57891bdc5c11721cb1a5b8fd9d093175a8f26c2d08d4c726f4b2907cf4863d
Oct 30 19:17:41.921693 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.921662    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:17:41.921780 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.921729    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:17:41.921780 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:41.921758    4924 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 19:17:42.054834 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.054797    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:42.164955 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.164918    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf" event={"ID":"99520a11-7547-4a41-90e6-f03de8d3cccd","Type":"ContainerStarted","Data":"c3f427456686751bc22d210a093a11e05e14e25313141fa5d59c2e7173163677"}
Oct 30 19:17:42.165204 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.164982    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf" event={"ID":"99520a11-7547-4a41-90e6-f03de8d3cccd","Type":"ContainerStarted","Data":"6f57891bdc5c11721cb1a5b8fd9d093175a8f26c2d08d4c726f4b2907cf4863d"}
Oct 30 19:17:42.166847 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.166770    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5" event={"ID":"9bbf86bc-d98c-4c76-b96e-ff9a48795800","Type":"ContainerStarted","Data":"0a0ca4b06f021572fe567e03033e89e7c78f981c6243d6d808f2d5f97f404ee9"}
Oct 30 19:17:42.166847 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.166812    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5" event={"ID":"9bbf86bc-d98c-4c76-b96e-ff9a48795800","Type":"ContainerStarted","Data":"68354f4da31b4784ad198138a77a0f0f985d26e9cdcbf4fca981004930cfdfa8"}
Oct 30 19:17:42.168430 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.168409    4924 generic.go:356] "Generic (PLEG): container finished" podID="4dab0473-fa74-460f-8b18-69874157c8fa" containerID="8d2c6380a2d90f3b4be291705554417d2ef61505b1c901eb7534f3058e715aef" exitCode=0
Oct 30 19:17:42.168562 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.168435    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv" event={"ID":"4dab0473-fa74-460f-8b18-69874157c8fa","Type":"ContainerDied","Data":"8d2c6380a2d90f3b4be291705554417d2ef61505b1c901eb7534f3058e715aef"}
Oct 30 19:17:42.168562 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.168451    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"
Oct 30 19:17:42.168562 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.168464    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv" event={"ID":"4dab0473-fa74-460f-8b18-69874157c8fa","Type":"ContainerDied","Data":"f62b9002716833944bccf44d27fc56de639a4b2c519710fdf154ab70aa7ac98f"}
Oct 30 19:17:42.168562 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.168490    4924 scope.go:122] "RemoveContainer" containerID="8d2c6380a2d90f3b4be291705554417d2ef61505b1c901eb7534f3058e715aef"
Oct 30 19:17:42.179023 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.179006    4924 scope.go:122] "RemoveContainer" containerID="8d2c6380a2d90f3b4be291705554417d2ef61505b1c901eb7534f3058e715aef"
Oct 30 19:17:42.179400 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:17:42.179380    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d2c6380a2d90f3b4be291705554417d2ef61505b1c901eb7534f3058e715aef\": container with ID starting with 8d2c6380a2d90f3b4be291705554417d2ef61505b1c901eb7534f3058e715aef not found: ID does not exist" containerID="8d2c6380a2d90f3b4be291705554417d2ef61505b1c901eb7534f3058e715aef"
Oct 30 19:17:42.179488 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.179405    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d2c6380a2d90f3b4be291705554417d2ef61505b1c901eb7534f3058e715aef"} err="failed to get container status \"8d2c6380a2d90f3b4be291705554417d2ef61505b1c901eb7534f3058e715aef\": rpc error: code = NotFound desc = could not find container \"8d2c6380a2d90f3b4be291705554417d2ef61505b1c901eb7534f3058e715aef\": container with ID starting with 8d2c6380a2d90f3b4be291705554417d2ef61505b1c901eb7534f3058e715aef not found: ID does not exist"
Oct 30 19:17:42.211989 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.211957    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld" podUID="f04af09f-b8d2-4b47-b0b3-88eb12adff93" containerName="main" containerID="cri-o://8df63936a23a9397c6a763772d2f3e8852effa7844bbaef26e08152f8dc3523b" gracePeriod=45
Oct 30 19:17:42.258711 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.258664    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m" podUID="1a90412d-c047-422e-b3c7-9546d286c303" containerName="main" containerID="cri-o://297bf6e960483b78655aeb436dc75535c47c8bec446e852da809dda6fe8dffd0" gracePeriod=45
Oct 30 19:17:42.281487 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.281422    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf" podStartSLOduration=1.281403262 podStartE2EDuration="1.281403262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:42.277952651 +0000 UTC m=+2463.555003107" watchObservedRunningTime="2026-10-30 19:17:42.281403262 +0000 UTC m=+2463.558453703"
Oct 30 19:17:42.390622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.390559    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5" podStartSLOduration=2.390542214 podStartE2EDuration="2.390542214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:42.389440036 +0000 UTC m=+2463.666490465" watchObservedRunningTime="2026-10-30 19:17:42.390542214 +0000 UTC m=+2463.667592660"
Oct 30 19:17:42.534203 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.534125    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"]
Oct 30 19:17:42.534617 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.534551    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:42.542456 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.542049    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-75e91b-openshift-default-5546f5b579-qggnv"]
Oct 30 19:17:42.557515 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.557147    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mh9pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Oct 30 19:17:42.557515 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.557214    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf" podUID="99520a11-7547-4a41-90e6-f03de8d3cccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Oct 30 19:17:42.579993 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.579968    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 19:17:42.601400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.601371    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 19:17:42.734530 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.733230    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a90412d-c047-422e-b3c7-9546d286c303-kube-api-access-drrgn\" (UniqueName: \"kubernetes.io/projected/1a90412d-c047-422e-b3c7-9546d286c303-kube-api-access-drrgn\") pod \"1a90412d-c047-422e-b3c7-9546d286c303\" (UID: \"1a90412d-c047-422e-b3c7-9546d286c303\") "
Oct 30 19:17:42.734530 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.733296    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f04af09f-b8d2-4b47-b0b3-88eb12adff93-tls-certs\" (UniqueName: \"kubernetes.io/secret/f04af09f-b8d2-4b47-b0b3-88eb12adff93-tls-certs\") pod \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\" (UID: \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\") "
Oct 30 19:17:42.734530 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.733342    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-tmp-dir\") pod \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\" (UID: \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\") "
Oct 30 19:17:42.734530 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.733449    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a90412d-c047-422e-b3c7-9546d286c303-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a90412d-c047-422e-b3c7-9546d286c303-tls-certs\") pod \"1a90412d-c047-422e-b3c7-9546d286c303\" (UID: \"1a90412d-c047-422e-b3c7-9546d286c303\") "
Oct 30 19:17:42.734530 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.733491    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-model-cache\") pod \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\" (UID: \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\") "
Oct 30 19:17:42.734530 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.733547    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-home\" (UniqueName: \"kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-home\") pod \"1a90412d-c047-422e-b3c7-9546d286c303\" (UID: \"1a90412d-c047-422e-b3c7-9546d286c303\") "
Oct 30 19:17:42.734530 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.733600    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-dshm\") pod \"1a90412d-c047-422e-b3c7-9546d286c303\" (UID: \"1a90412d-c047-422e-b3c7-9546d286c303\") "
Oct 30 19:17:42.734530 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.733677    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-tmp-dir\") pod \"1a90412d-c047-422e-b3c7-9546d286c303\" (UID: \"1a90412d-c047-422e-b3c7-9546d286c303\") "
Oct 30 19:17:42.734530 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.733706    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-home\" (UniqueName: \"kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-home\") pod \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\" (UID: \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\") "
Oct 30 19:17:42.734530 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.733737    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-dshm\") pod \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\" (UID: \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\") "
Oct 30 19:17:42.734530 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.733768    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f04af09f-b8d2-4b47-b0b3-88eb12adff93-kube-api-access-kbhg8\" (UniqueName: \"kubernetes.io/projected/f04af09f-b8d2-4b47-b0b3-88eb12adff93-kube-api-access-kbhg8\") pod \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\" (UID: \"f04af09f-b8d2-4b47-b0b3-88eb12adff93\") "
Oct 30 19:17:42.734530 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.733796    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-model-cache\") pod \"1a90412d-c047-422e-b3c7-9546d286c303\" (UID: \"1a90412d-c047-422e-b3c7-9546d286c303\") "
Oct 30 19:17:42.740519 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.737195    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-model-cache" pod "1a90412d-c047-422e-b3c7-9546d286c303" (UID: "1a90412d-c047-422e-b3c7-9546d286c303"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:42.740519 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.738175    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-tmp-dir" pod "f04af09f-b8d2-4b47-b0b3-88eb12adff93" (UID: "f04af09f-b8d2-4b47-b0b3-88eb12adff93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:42.740519 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.739217    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-tmp-dir" pod "1a90412d-c047-422e-b3c7-9546d286c303" (UID: "1a90412d-c047-422e-b3c7-9546d286c303"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:42.740519 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.739509    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-home" pod "f04af09f-b8d2-4b47-b0b3-88eb12adff93" (UID: "f04af09f-b8d2-4b47-b0b3-88eb12adff93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:42.740519 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.739995    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-model-cache" pod "f04af09f-b8d2-4b47-b0b3-88eb12adff93" (UID: "f04af09f-b8d2-4b47-b0b3-88eb12adff93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:42.740519 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.740202    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-home" pod "1a90412d-c047-422e-b3c7-9546d286c303" (UID: "1a90412d-c047-422e-b3c7-9546d286c303"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:42.743338 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.743296    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-dshm" pod "1a90412d-c047-422e-b3c7-9546d286c303" (UID: "1a90412d-c047-422e-b3c7-9546d286c303"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:42.744270 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.744236    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-dshm" pod "f04af09f-b8d2-4b47-b0b3-88eb12adff93" (UID: "f04af09f-b8d2-4b47-b0b3-88eb12adff93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:42.744995 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.744960    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f04af09f-b8d2-4b47-b0b3-88eb12adff93-kube-api-access-kbhg8" pod "f04af09f-b8d2-4b47-b0b3-88eb12adff93" (UID: "f04af09f-b8d2-4b47-b0b3-88eb12adff93"). InnerVolumeSpecName "kube-api-access-kbhg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:42.747489 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.747447    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a90412d-c047-422e-b3c7-9546d286c303-tls-certs" pod "1a90412d-c047-422e-b3c7-9546d286c303" (UID: "1a90412d-c047-422e-b3c7-9546d286c303"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:42.747767 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.747730    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a90412d-c047-422e-b3c7-9546d286c303-kube-api-access-drrgn" pod "1a90412d-c047-422e-b3c7-9546d286c303" (UID: "1a90412d-c047-422e-b3c7-9546d286c303"). InnerVolumeSpecName "kube-api-access-drrgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:42.748974 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.748953    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f04af09f-b8d2-4b47-b0b3-88eb12adff93-tls-certs" pod "f04af09f-b8d2-4b47-b0b3-88eb12adff93" (UID: "f04af09f-b8d2-4b47-b0b3-88eb12adff93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:42.759254 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.759229    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 19:17:42.839089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.837188    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/467dd97e-420c-44ac-a249-17a3deacfb63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467dd97e-420c-44ac-a249-17a3deacfb63-istiod-ca-cert\") pod \"467dd97e-420c-44ac-a249-17a3deacfb63\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") "
Oct 30 19:17:42.839089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.837248    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467dd97e-420c-44ac-a249-17a3deacfb63-istio-token\" (UniqueName: \"kubernetes.io/projected/467dd97e-420c-44ac-a249-17a3deacfb63-istio-token\") pod \"467dd97e-420c-44ac-a249-17a3deacfb63\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") "
Oct 30 19:17:42.839089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.837325    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-workload-socket\") pod \"467dd97e-420c-44ac-a249-17a3deacfb63\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") "
Oct 30 19:17:42.839089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.837369    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/467dd97e-420c-44ac-a249-17a3deacfb63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467dd97e-420c-44ac-a249-17a3deacfb63-istio-podinfo\") pod \"467dd97e-420c-44ac-a249-17a3deacfb63\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") "
Oct 30 19:17:42.839089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.837420    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467dd97e-420c-44ac-a249-17a3deacfb63-kube-api-access-lzhtl\" (UniqueName: \"kubernetes.io/projected/467dd97e-420c-44ac-a249-17a3deacfb63-kube-api-access-lzhtl\") pod \"467dd97e-420c-44ac-a249-17a3deacfb63\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") "
Oct 30 19:17:42.839089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.837479    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-credential-socket\") pod \"467dd97e-420c-44ac-a249-17a3deacfb63\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") "
Oct 30 19:17:42.839089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.837522    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-istio-envoy\") pod \"467dd97e-420c-44ac-a249-17a3deacfb63\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") "
Oct 30 19:17:42.839089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.837568    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-istio-data\") pod \"467dd97e-420c-44ac-a249-17a3deacfb63\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") "
Oct 30 19:17:42.839089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.837654    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-workload-certs\") pod \"467dd97e-420c-44ac-a249-17a3deacfb63\" (UID: \"467dd97e-420c-44ac-a249-17a3deacfb63\") "
Oct 30 19:17:42.839089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.838025    4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-tmp-dir\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.839089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.838045    4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a90412d-c047-422e-b3c7-9546d286c303-tls-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.839089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.838057    4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-model-cache\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.839089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.838087    4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-home\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.839089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.838100    4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-dshm\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.839089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.838113    4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-tmp-dir\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.839089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.838125    4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-home\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.839089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.838135    4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f04af09f-b8d2-4b47-b0b3-88eb12adff93-dshm\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.839089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.838148    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbhg8\" (UniqueName: \"kubernetes.io/projected/f04af09f-b8d2-4b47-b0b3-88eb12adff93-kube-api-access-kbhg8\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.839089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.838161    4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a90412d-c047-422e-b3c7-9546d286c303-model-cache\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.839089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.838173    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drrgn\" (UniqueName: \"kubernetes.io/projected/1a90412d-c047-422e-b3c7-9546d286c303-kube-api-access-drrgn\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.839089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.838185    4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f04af09f-b8d2-4b47-b0b3-88eb12adff93-tls-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.839089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.838438    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-workload-certs" pod "467dd97e-420c-44ac-a249-17a3deacfb63" (UID: "467dd97e-420c-44ac-a249-17a3deacfb63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:42.839089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.838795    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/467dd97e-420c-44ac-a249-17a3deacfb63-istiod-ca-cert" pod "467dd97e-420c-44ac-a249-17a3deacfb63" (UID: "467dd97e-420c-44ac-a249-17a3deacfb63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:42.839089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.839032    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-credential-socket" pod "467dd97e-420c-44ac-a249-17a3deacfb63" (UID: "467dd97e-420c-44ac-a249-17a3deacfb63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:42.840559 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.839608    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-istio-data" pod "467dd97e-420c-44ac-a249-17a3deacfb63" (UID: "467dd97e-420c-44ac-a249-17a3deacfb63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:42.840559 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.839830    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-workload-socket" pod "467dd97e-420c-44ac-a249-17a3deacfb63" (UID: "467dd97e-420c-44ac-a249-17a3deacfb63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:42.842987 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.842958    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467dd97e-420c-44ac-a249-17a3deacfb63-kube-api-access-lzhtl" pod "467dd97e-420c-44ac-a249-17a3deacfb63" (UID: "467dd97e-420c-44ac-a249-17a3deacfb63"). InnerVolumeSpecName "kube-api-access-lzhtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:42.843200 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.843175    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/467dd97e-420c-44ac-a249-17a3deacfb63-istio-podinfo" pod "467dd97e-420c-44ac-a249-17a3deacfb63" (UID: "467dd97e-420c-44ac-a249-17a3deacfb63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:17:42.843325 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.843203    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467dd97e-420c-44ac-a249-17a3deacfb63-istio-token" pod "467dd97e-420c-44ac-a249-17a3deacfb63" (UID: "467dd97e-420c-44ac-a249-17a3deacfb63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:42.844387 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.844362    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-istio-envoy" pod "467dd97e-420c-44ac-a249-17a3deacfb63" (UID: "467dd97e-420c-44ac-a249-17a3deacfb63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:42.846208 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.846177    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dab0473-fa74-460f-8b18-69874157c8fa" path="/var/lib/kubelet/pods/4dab0473-fa74-460f-8b18-69874157c8fa/volumes"
Oct 30 19:17:42.939020 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.938977    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzhtl\" (UniqueName: \"kubernetes.io/projected/467dd97e-420c-44ac-a249-17a3deacfb63-kube-api-access-lzhtl\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.939020 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.939020    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.939568 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.939035    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.939568 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.939049    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.939568 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.939097    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.939568 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.939113    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467dd97e-420c-44ac-a249-17a3deacfb63-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.939568 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.939126    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467dd97e-420c-44ac-a249-17a3deacfb63-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.939568 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.939142    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467dd97e-420c-44ac-a249-17a3deacfb63-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:42.939568 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:42.939156    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467dd97e-420c-44ac-a249-17a3deacfb63-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:43.094509 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.094478    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"
Oct 30 19:17:43.175752 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.175718    4924 generic.go:356] "Generic (PLEG): container finished" podID="f04af09f-b8d2-4b47-b0b3-88eb12adff93" containerID="8df63936a23a9397c6a763772d2f3e8852effa7844bbaef26e08152f8dc3523b" exitCode=0
Oct 30 19:17:43.175935 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.175766    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"
Oct 30 19:17:43.175935 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.175803    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld" event={"ID":"f04af09f-b8d2-4b47-b0b3-88eb12adff93","Type":"ContainerDied","Data":"8df63936a23a9397c6a763772d2f3e8852effa7844bbaef26e08152f8dc3523b"}
Oct 30 19:17:43.175935 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.175851    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld" event={"ID":"f04af09f-b8d2-4b47-b0b3-88eb12adff93","Type":"ContainerDied","Data":"08a62a56a121b9500a464e8f396f20f26e97af55037d97599bdbf298ffe22312"}
Oct 30 19:17:43.175935 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.175873    4924 scope.go:122] "RemoveContainer" containerID="8df63936a23a9397c6a763772d2f3e8852effa7844bbaef26e08152f8dc3523b"
Oct 30 19:17:43.178522 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.178497    4924 generic.go:356] "Generic (PLEG): container finished" podID="467dd97e-420c-44ac-a249-17a3deacfb63" containerID="de74ea692d63df6c2372d51537836a2f9171a9a0890b72e08b0506c6818fd512" exitCode=0
Oct 30 19:17:43.178630 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.178561    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"
Oct 30 19:17:43.178630 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.178618    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz" event={"ID":"467dd97e-420c-44ac-a249-17a3deacfb63","Type":"ContainerDied","Data":"de74ea692d63df6c2372d51537836a2f9171a9a0890b72e08b0506c6818fd512"}
Oct 30 19:17:43.178735 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.178656    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz" event={"ID":"467dd97e-420c-44ac-a249-17a3deacfb63","Type":"ContainerDied","Data":"6d442f3484aad27339470a68d59135f628f84240d60b80f9007598cb973a4e85"}
Oct 30 19:17:43.180105 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.180084    4924 generic.go:356] "Generic (PLEG): container finished" podID="1a90412d-c047-422e-b3c7-9546d286c303" containerID="297bf6e960483b78655aeb436dc75535c47c8bec446e852da809dda6fe8dffd0" exitCode=0
Oct 30 19:17:43.180216 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.180142    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"
Oct 30 19:17:43.180216 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.180174    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m" event={"ID":"1a90412d-c047-422e-b3c7-9546d286c303","Type":"ContainerDied","Data":"297bf6e960483b78655aeb436dc75535c47c8bec446e852da809dda6fe8dffd0"}
Oct 30 19:17:43.180216 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.180201    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m" event={"ID":"1a90412d-c047-422e-b3c7-9546d286c303","Type":"ContainerDied","Data":"51267a37bad47f3b51bdf3f01cdb3c1d281b39d040665c3f60f78fb134fcb916"}
Oct 30 19:17:43.186511 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.186494    4924 scope.go:122] "RemoveContainer" containerID="8df63936a23a9397c6a763772d2f3e8852effa7844bbaef26e08152f8dc3523b"
Oct 30 19:17:43.186779 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:17:43.186763    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8df63936a23a9397c6a763772d2f3e8852effa7844bbaef26e08152f8dc3523b\": container with ID starting with 8df63936a23a9397c6a763772d2f3e8852effa7844bbaef26e08152f8dc3523b not found: ID does not exist" containerID="8df63936a23a9397c6a763772d2f3e8852effa7844bbaef26e08152f8dc3523b"
Oct 30 19:17:43.186836 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.186785    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8df63936a23a9397c6a763772d2f3e8852effa7844bbaef26e08152f8dc3523b"} err="failed to get container status \"8df63936a23a9397c6a763772d2f3e8852effa7844bbaef26e08152f8dc3523b\": rpc error: code = NotFound desc = could not find container \"8df63936a23a9397c6a763772d2f3e8852effa7844bbaef26e08152f8dc3523b\": container with ID starting with 8df63936a23a9397c6a763772d2f3e8852effa7844bbaef26e08152f8dc3523b not found: ID does not exist"
Oct 30 19:17:43.186836 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.186802    4924 scope.go:122] "RemoveContainer" containerID="de74ea692d63df6c2372d51537836a2f9171a9a0890b72e08b0506c6818fd512"
Oct 30 19:17:43.195729 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.195712    4924 scope.go:122] "RemoveContainer" containerID="de74ea692d63df6c2372d51537836a2f9171a9a0890b72e08b0506c6818fd512"
Oct 30 19:17:43.195969 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:17:43.195951    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de74ea692d63df6c2372d51537836a2f9171a9a0890b72e08b0506c6818fd512\": container with ID starting with de74ea692d63df6c2372d51537836a2f9171a9a0890b72e08b0506c6818fd512 not found: ID does not exist" containerID="de74ea692d63df6c2372d51537836a2f9171a9a0890b72e08b0506c6818fd512"
Oct 30 19:17:43.196036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.195977    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de74ea692d63df6c2372d51537836a2f9171a9a0890b72e08b0506c6818fd512"} err="failed to get container status \"de74ea692d63df6c2372d51537836a2f9171a9a0890b72e08b0506c6818fd512\": rpc error: code = NotFound desc = could not find container \"de74ea692d63df6c2372d51537836a2f9171a9a0890b72e08b0506c6818fd512\": container with ID starting with de74ea692d63df6c2372d51537836a2f9171a9a0890b72e08b0506c6818fd512 not found: ID does not exist"
Oct 30 19:17:43.196036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.195992    4924 scope.go:122] "RemoveContainer" containerID="297bf6e960483b78655aeb436dc75535c47c8bec446e852da809dda6fe8dffd0"
Oct 30 19:17:43.229171 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.229148    4924 scope.go:122] "RemoveContainer" containerID="297bf6e960483b78655aeb436dc75535c47c8bec446e852da809dda6fe8dffd0"
Oct 30 19:17:43.229587 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:17:43.229565    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"297bf6e960483b78655aeb436dc75535c47c8bec446e852da809dda6fe8dffd0\": container with ID starting with 297bf6e960483b78655aeb436dc75535c47c8bec446e852da809dda6fe8dffd0 not found: ID does not exist" containerID="297bf6e960483b78655aeb436dc75535c47c8bec446e852da809dda6fe8dffd0"
Oct 30 19:17:43.229662 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.229596    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"297bf6e960483b78655aeb436dc75535c47c8bec446e852da809dda6fe8dffd0"} err="failed to get container status \"297bf6e960483b78655aeb436dc75535c47c8bec446e852da809dda6fe8dffd0\": rpc error: code = NotFound desc = could not find container \"297bf6e960483b78655aeb436dc75535c47c8bec446e852da809dda6fe8dffd0\": container with ID starting with 297bf6e960483b78655aeb436dc75535c47c8bec446e852da809dda6fe8dffd0 not found: ID does not exist"
Oct 30 19:17:43.338248 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.338143    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"]
Oct 30 19:17:43.384597 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.384560    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec8dab-openshift-default-6948fb4f79-mf4tz"]
Oct 30 19:17:43.449029 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.448993    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"]
Oct 30 19:17:43.453703 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.453674    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-3b6ad6/distinct-w5-3b6ad6-kserve-94bc474-mbpld"]
Oct 30 19:17:43.533591 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.533551    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mh9pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Oct 30 19:17:43.533765 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.533607    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf" podUID="99520a11-7547-4a41-90e6-f03de8d3cccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Oct 30 19:17:43.558823 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.558792    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"]
Oct 30 19:17:43.561382 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:43.561358    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-3b6ad6/simulator-w5-3b6ad6-kserve-84797f4658-8sj7m"]
Oct 30 19:17:44.533199 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:44.533165    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mh9pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Oct 30 19:17:44.533607 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:44.533220    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf" podUID="99520a11-7547-4a41-90e6-f03de8d3cccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Oct 30 19:17:44.836536 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:44.836460    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a90412d-c047-422e-b3c7-9546d286c303" path="/var/lib/kubelet/pods/1a90412d-c047-422e-b3c7-9546d286c303/volumes"
Oct 30 19:17:44.836944 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:44.836931    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="467dd97e-420c-44ac-a249-17a3deacfb63" path="/var/lib/kubelet/pods/467dd97e-420c-44ac-a249-17a3deacfb63/volumes"
Oct 30 19:17:44.837342 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:44.837328    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f04af09f-b8d2-4b47-b0b3-88eb12adff93" path="/var/lib/kubelet/pods/f04af09f-b8d2-4b47-b0b3-88eb12adff93/volumes"
Oct 30 19:17:44.891568 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:44.891537    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"]
Oct 30 19:17:44.891982 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:44.891969    4924 cpu_manager.go:414] "containerMap: removing container" podUID="4dab0473-fa74-460f-8b18-69874157c8fa" containerName="istio-proxy"
Oct 30 19:17:44.892040 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:44.891984    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dab0473-fa74-460f-8b18-69874157c8fa" containerName="istio-proxy"
Oct 30 19:17:44.892040 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:44.892000    4924 cpu_manager.go:414] "containerMap: removing container" podUID="f04af09f-b8d2-4b47-b0b3-88eb12adff93" containerName="main"
Oct 30 19:17:44.892040 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:44.892005    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f04af09f-b8d2-4b47-b0b3-88eb12adff93" containerName="main"
Oct 30 19:17:44.892040 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:44.892026    4924 cpu_manager.go:414] "containerMap: removing container" podUID="1a90412d-c047-422e-b3c7-9546d286c303" containerName="main"
Oct 30 19:17:44.892040 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:44.892030    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a90412d-c047-422e-b3c7-9546d286c303" containerName="main"
Oct 30 19:17:44.892238 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:44.892048    4924 cpu_manager.go:414] "containerMap: removing container" podUID="467dd97e-420c-44ac-a249-17a3deacfb63" containerName="istio-proxy"
Oct 30 19:17:44.892238 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:44.892053    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467dd97e-420c-44ac-a249-17a3deacfb63" containerName="istio-proxy"
Oct 30 19:17:44.892238 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:44.892154    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="467dd97e-420c-44ac-a249-17a3deacfb63" containerName="istio-proxy"
Oct 30 19:17:44.892238 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:44.892167    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dab0473-fa74-460f-8b18-69874157c8fa" containerName="istio-proxy"
Oct 30 19:17:44.892238 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:44.892177    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="f04af09f-b8d2-4b47-b0b3-88eb12adff93" containerName="main"
Oct 30 19:17:44.892238 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:44.892191    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a90412d-c047-422e-b3c7-9546d286c303" containerName="main"
Oct 30 19:17:44.927346 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:44.927312    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"
Oct 30 19:17:44.956830 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:44.956798    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"]
Oct 30 19:17:44.977583 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:44.977557    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-29de7b\"/\"test-model-29de7b-kserve-self-signed-certs\""
Oct 30 19:17:44.982993 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:44.982955    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:44.998453 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:44.998425    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-29de7b\"/\"default-dockercfg-chzfp\""
Oct 30 19:17:44.998577 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:44.998465    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-29de7b\"/\"openshift-service-ca.crt\""
Oct 30 19:17:44.998577 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:44.998504    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-29de7b\"/\"kube-root-ca.crt\""
Oct 30 19:17:45.062709 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:45.062671    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5td4r\" (UniqueName: \"kubernetes.io/projected/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-kube-api-access-5td4r\") pod \"test-model-29de7b-kserve-5798d9fcbf-vssfw\" (UID: \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\") " pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"
Oct 30 19:17:45.062909 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:45.062714    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-home\") pod \"test-model-29de7b-kserve-5798d9fcbf-vssfw\" (UID: \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\") " pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"
Oct 30 19:17:45.062909 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:45.062745    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-tls-certs\") pod \"test-model-29de7b-kserve-5798d9fcbf-vssfw\" (UID: \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\") " pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"
Oct 30 19:17:45.062909 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:45.062878    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-tmp-dir\") pod \"test-model-29de7b-kserve-5798d9fcbf-vssfw\" (UID: \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\") " pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"
Oct 30 19:17:45.063030 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:45.062914    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-dshm\") pod \"test-model-29de7b-kserve-5798d9fcbf-vssfw\" (UID: \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\") " pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"
Oct 30 19:17:45.063030 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:45.062974    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-model-cache\") pod \"test-model-29de7b-kserve-5798d9fcbf-vssfw\" (UID: \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\") " pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"
Oct 30 19:17:45.164384 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:45.164336    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5td4r\" (UniqueName: \"kubernetes.io/projected/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-kube-api-access-5td4r\") pod \"test-model-29de7b-kserve-5798d9fcbf-vssfw\" (UID: \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\") " pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"
Oct 30 19:17:45.164384 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:45.164387    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-home\") pod \"test-model-29de7b-kserve-5798d9fcbf-vssfw\" (UID: \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\") " pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"
Oct 30 19:17:45.164676 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:45.164422    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-tls-certs\") pod \"test-model-29de7b-kserve-5798d9fcbf-vssfw\" (UID: \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\") " pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"
Oct 30 19:17:45.164676 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:45.164476    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-tmp-dir\") pod \"test-model-29de7b-kserve-5798d9fcbf-vssfw\" (UID: \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\") " pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"
Oct 30 19:17:45.164676 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:45.164506    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-dshm\") pod \"test-model-29de7b-kserve-5798d9fcbf-vssfw\" (UID: \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\") " pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"
Oct 30 19:17:45.164676 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:45.164583    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-model-cache\") pod \"test-model-29de7b-kserve-5798d9fcbf-vssfw\" (UID: \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\") " pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"
Oct 30 19:17:45.164890 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:45.164868    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-home\") pod \"test-model-29de7b-kserve-5798d9fcbf-vssfw\" (UID: \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\") " pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"
Oct 30 19:17:45.164951 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:45.164926    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-tmp-dir\") pod \"test-model-29de7b-kserve-5798d9fcbf-vssfw\" (UID: \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\") " pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"
Oct 30 19:17:45.165005 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:45.164989    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-model-cache\") pod \"test-model-29de7b-kserve-5798d9fcbf-vssfw\" (UID: \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\") " pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"
Oct 30 19:17:45.166972 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:45.166948    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-dshm\") pod \"test-model-29de7b-kserve-5798d9fcbf-vssfw\" (UID: \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\") " pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"
Oct 30 19:17:45.167142 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:45.167126    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-tls-certs\") pod \"test-model-29de7b-kserve-5798d9fcbf-vssfw\" (UID: \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\") " pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"
Oct 30 19:17:45.227828 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:45.227790    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5td4r\" (UniqueName: \"kubernetes.io/projected/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-kube-api-access-5td4r\") pod \"test-model-29de7b-kserve-5798d9fcbf-vssfw\" (UID: \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\") " pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"
Oct 30 19:17:45.248097 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:45.248050    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"
Oct 30 19:17:45.394989 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:45.394957    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:45.533788 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:45.533620    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mh9pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Oct 30 19:17:45.533788 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:45.533684    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf" podUID="99520a11-7547-4a41-90e6-f03de8d3cccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Oct 30 19:17:45.661005 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:45.660973    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"]
Oct 30 19:17:45.661342 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:17:45.661318    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad5175c4_eeb1_4f0f_a8c0_12897baa3b4f.slice/crio-c4adfb30159ef100cbdb6ec6b7c04d413777a974a6633d4f6919fe22cc5bafbe WatchSource:0}: Error finding container c4adfb30159ef100cbdb6ec6b7c04d413777a974a6633d4f6919fe22cc5bafbe: Status 404 returned error can't find the container with id c4adfb30159ef100cbdb6ec6b7c04d413777a974a6633d4f6919fe22cc5bafbe
Oct 30 19:17:45.737474 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:17:45.737439    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:17:45.782965 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:45.782934    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:46.228799 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:46.228760    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw" event={"ID":"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f","Type":"ContainerStarted","Data":"066df397d77cff5c3b219cbd8fda2f0ab33d6a18972d13daaa5925a17563f626"}
Oct 30 19:17:46.228983 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:46.228812    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw" event={"ID":"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f","Type":"ContainerStarted","Data":"c4adfb30159ef100cbdb6ec6b7c04d413777a974a6633d4f6919fe22cc5bafbe"}
Oct 30 19:17:46.249082 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:46.249039    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"
Oct 30 19:17:46.338672 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:46.338620    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw" podStartSLOduration=2.338605666 podStartE2EDuration="2.338605666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:46.337739436 +0000 UTC m=+2467.614789881" watchObservedRunningTime="2026-10-30 19:17:46.338605666 +0000 UTC m=+2467.615656111"
Oct 30 19:17:46.533495 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:46.533399    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mh9pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Oct 30 19:17:46.533495 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:46.533472    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf" podUID="99520a11-7547-4a41-90e6-f03de8d3cccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Oct 30 19:17:47.533666 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:47.533629    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mh9pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Oct 30 19:17:47.534082 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:47.533686    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf" podUID="99520a11-7547-4a41-90e6-f03de8d3cccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Oct 30 19:17:48.534148 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:48.534104    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mh9pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Oct 30 19:17:48.534666 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:48.534170    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf" podUID="99520a11-7547-4a41-90e6-f03de8d3cccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Oct 30 19:17:49.466394 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:49.466358    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq" podUID="539e8daa-2f24-4d43-8017-237fb777d90e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:17:49.534247 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:49.534207    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mh9pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Oct 30 19:17:49.534741 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:49.534256    4924 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf" podUID="99520a11-7547-4a41-90e6-f03de8d3cccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Oct 30 19:17:50.089827 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:50.089788    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:50.418673 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:50.418636    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:50.537617 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:50.537585    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:50.538104 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:50.538090    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:50.538754 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:50.538734    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mh9pf"
Oct 30 19:17:51.094369 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:51.094338    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"
Oct 30 19:17:51.107137 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:51.107112    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"
Oct 30 19:17:51.280314 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:51.280287    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"
Oct 30 19:17:55.248252 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:55.248217    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"
Oct 30 19:17:55.260991 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:55.260968    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"
Oct 30 19:17:55.267086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:55.267036    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:55.306283 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:55.306257    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"
Oct 30 19:17:59.246285 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:59.246225    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq"
Oct 30 19:17:59.321789 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:59.321757    4924 generic.go:356] "Generic (PLEG): container finished" podID="539e8daa-2f24-4d43-8017-237fb777d90e" containerID="d8cbacada5c40930e6550b79d44fc1752c4ec5f2df7884738014c20e0cc0eb04" exitCode=0
Oct 30 19:17:59.321936 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:59.321801    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq"
Oct 30 19:17:59.321936 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:59.321842    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq" event={"ID":"539e8daa-2f24-4d43-8017-237fb777d90e","Type":"ContainerDied","Data":"d8cbacada5c40930e6550b79d44fc1752c4ec5f2df7884738014c20e0cc0eb04"}
Oct 30 19:17:59.321936 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:59.321892    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq" event={"ID":"539e8daa-2f24-4d43-8017-237fb777d90e","Type":"ContainerDied","Data":"1efda1fdef41aff662e850beee251c2c2096591b3f9935ab03014ee16f9b038c"}
Oct 30 19:17:59.321936 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:59.321921    4924 scope.go:122] "RemoveContainer" containerID="d8cbacada5c40930e6550b79d44fc1752c4ec5f2df7884738014c20e0cc0eb04"
Oct 30 19:17:59.328332 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:59.328304    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/539e8daa-2f24-4d43-8017-237fb777d90e-kube-api-access-qccdh\" (UniqueName: \"kubernetes.io/projected/539e8daa-2f24-4d43-8017-237fb777d90e-kube-api-access-qccdh\") pod \"539e8daa-2f24-4d43-8017-237fb777d90e\" (UID: \"539e8daa-2f24-4d43-8017-237fb777d90e\") "
Oct 30 19:17:59.328450 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:59.328356    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/539e8daa-2f24-4d43-8017-237fb777d90e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/539e8daa-2f24-4d43-8017-237fb777d90e-plugins-config-volume\") pod \"539e8daa-2f24-4d43-8017-237fb777d90e\" (UID: \"539e8daa-2f24-4d43-8017-237fb777d90e\") "
Oct 30 19:17:59.328704 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:59.328685    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/539e8daa-2f24-4d43-8017-237fb777d90e-plugins-config-volume" pod "539e8daa-2f24-4d43-8017-237fb777d90e" (UID: "539e8daa-2f24-4d43-8017-237fb777d90e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:59.330374 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:59.330351    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/539e8daa-2f24-4d43-8017-237fb777d90e-kube-api-access-qccdh" pod "539e8daa-2f24-4d43-8017-237fb777d90e" (UID: "539e8daa-2f24-4d43-8017-237fb777d90e"). InnerVolumeSpecName "kube-api-access-qccdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:59.331784 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:59.331764    4924 scope.go:122] "RemoveContainer" containerID="d8cbacada5c40930e6550b79d44fc1752c4ec5f2df7884738014c20e0cc0eb04"
Oct 30 19:17:59.332059 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:17:59.332034    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8cbacada5c40930e6550b79d44fc1752c4ec5f2df7884738014c20e0cc0eb04\": container with ID starting with d8cbacada5c40930e6550b79d44fc1752c4ec5f2df7884738014c20e0cc0eb04 not found: ID does not exist" containerID="d8cbacada5c40930e6550b79d44fc1752c4ec5f2df7884738014c20e0cc0eb04"
Oct 30 19:17:59.332146 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:59.332085    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8cbacada5c40930e6550b79d44fc1752c4ec5f2df7884738014c20e0cc0eb04"} err="failed to get container status \"d8cbacada5c40930e6550b79d44fc1752c4ec5f2df7884738014c20e0cc0eb04\": rpc error: code = NotFound desc = could not find container \"d8cbacada5c40930e6550b79d44fc1752c4ec5f2df7884738014c20e0cc0eb04\": container with ID starting with d8cbacada5c40930e6550b79d44fc1752c4ec5f2df7884738014c20e0cc0eb04 not found: ID does not exist"
Oct 30 19:17:59.429833 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:59.429790    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qccdh\" (UniqueName: \"kubernetes.io/projected/539e8daa-2f24-4d43-8017-237fb777d90e-kube-api-access-qccdh\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:59.429833 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:59.429822    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/539e8daa-2f24-4d43-8017-237fb777d90e-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:59.522538 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:59.522457    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:17:59.742394 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:59.742352    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq"]
Oct 30 19:17:59.747392 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:17:59.747365    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-3b6ad6-67448fd99f-zjgsq"]
Oct 30 19:18:00.841929 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:00.841893    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="539e8daa-2f24-4d43-8017-237fb777d90e" path="/var/lib/kubelet/pods/539e8daa-2f24-4d43-8017-237fb777d90e/volumes"
Oct 30 19:18:01.155742 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:01.155681    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:01.533789 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:01.533643    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:01.625227 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:01.625193    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql"]
Oct 30 19:18:01.671337 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:01.671304    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3b6ad6-lzfql"]
Oct 30 19:18:02.093789 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:02.093754    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:02.834339 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:02.834304    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80c4bbbc-11d1-40a4-9def-e6d079a72a44" path="/var/lib/kubelet/pods/80c4bbbc-11d1-40a4-9def-e6d079a72a44/volumes"
Oct 30 19:18:04.651164 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:04.651125    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:04.955712 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:04.955625    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:05.560135 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:05.560100    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:06.663535 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:06.663497    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:08.564421 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:08.564386    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:08.902992 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:08.902960    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:08.970805 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:08.970772    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"]
Oct 30 19:18:08.971007 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:08.970983    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f" podUID="1e73c307-868e-40e8-ad04-099a7446ccef" containerName="istio-proxy" containerID="cri-o://3652bb640212f6cdd25b148d97e83be90cf657b4972e7f0ae17815306ad84c62" gracePeriod=30
Oct 30 19:18:09.303566 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:09.303481    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:11.601138 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:11.601107    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:13.951202 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:13.951167    4924 patch_prober.go:29] interesting pod/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:18:13.951606 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:13.951229    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f" podUID="1e73c307-868e-40e8-ad04-099a7446ccef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:18:14.220594 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.220569    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 19:18:14.299648 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.299620    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e73c307-868e-40e8-ad04-099a7446ccef-istio-token\" (UniqueName: \"kubernetes.io/projected/1e73c307-868e-40e8-ad04-099a7446ccef-istio-token\") pod \"1e73c307-868e-40e8-ad04-099a7446ccef\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") "
Oct 30 19:18:14.299853 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.299656    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-workload-socket\") pod \"1e73c307-868e-40e8-ad04-099a7446ccef\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") "
Oct 30 19:18:14.299853 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.299684    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e73c307-868e-40e8-ad04-099a7446ccef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e73c307-868e-40e8-ad04-099a7446ccef-istiod-ca-cert\") pod \"1e73c307-868e-40e8-ad04-099a7446ccef\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") "
Oct 30 19:18:14.299853 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.299726    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e73c307-868e-40e8-ad04-099a7446ccef-kube-api-access-fj76d\" (UniqueName: \"kubernetes.io/projected/1e73c307-868e-40e8-ad04-099a7446ccef-kube-api-access-fj76d\") pod \"1e73c307-868e-40e8-ad04-099a7446ccef\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") "
Oct 30 19:18:14.299853 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.299749    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-istio-envoy\") pod \"1e73c307-868e-40e8-ad04-099a7446ccef\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") "
Oct 30 19:18:14.299853 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.299770    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-credential-socket\") pod \"1e73c307-868e-40e8-ad04-099a7446ccef\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") "
Oct 30 19:18:14.299853 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.299829    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-istio-data\") pod \"1e73c307-868e-40e8-ad04-099a7446ccef\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") "
Oct 30 19:18:14.300221 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.299893    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e73c307-868e-40e8-ad04-099a7446ccef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e73c307-868e-40e8-ad04-099a7446ccef-istio-podinfo\") pod \"1e73c307-868e-40e8-ad04-099a7446ccef\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") "
Oct 30 19:18:14.300221 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.299928    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-workload-socket" pod "1e73c307-868e-40e8-ad04-099a7446ccef" (UID: "1e73c307-868e-40e8-ad04-099a7446ccef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:14.300221 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.299935    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-workload-certs\") pod \"1e73c307-868e-40e8-ad04-099a7446ccef\" (UID: \"1e73c307-868e-40e8-ad04-099a7446ccef\") "
Oct 30 19:18:14.300471 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.300431    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-istio-data" pod "1e73c307-868e-40e8-ad04-099a7446ccef" (UID: "1e73c307-868e-40e8-ad04-099a7446ccef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:14.300471 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.300448    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-workload-certs" pod "1e73c307-868e-40e8-ad04-099a7446ccef" (UID: "1e73c307-868e-40e8-ad04-099a7446ccef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:14.300610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.300566    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:14.300668 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.300649    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-credential-socket" pod "1e73c307-868e-40e8-ad04-099a7446ccef" (UID: "1e73c307-868e-40e8-ad04-099a7446ccef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:14.300723 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.300693    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e73c307-868e-40e8-ad04-099a7446ccef-istiod-ca-cert" pod "1e73c307-868e-40e8-ad04-099a7446ccef" (UID: "1e73c307-868e-40e8-ad04-099a7446ccef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:18:14.302207 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.302176    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-istio-envoy" pod "1e73c307-868e-40e8-ad04-099a7446ccef" (UID: "1e73c307-868e-40e8-ad04-099a7446ccef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:18:14.302460 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.302430    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e73c307-868e-40e8-ad04-099a7446ccef-kube-api-access-fj76d" pod "1e73c307-868e-40e8-ad04-099a7446ccef" (UID: "1e73c307-868e-40e8-ad04-099a7446ccef"). InnerVolumeSpecName "kube-api-access-fj76d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:14.302574 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.302522    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e73c307-868e-40e8-ad04-099a7446ccef-istio-podinfo" pod "1e73c307-868e-40e8-ad04-099a7446ccef" (UID: "1e73c307-868e-40e8-ad04-099a7446ccef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:18:14.302574 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.302523    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e73c307-868e-40e8-ad04-099a7446ccef-istio-token" pod "1e73c307-868e-40e8-ad04-099a7446ccef" (UID: "1e73c307-868e-40e8-ad04-099a7446ccef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:14.401908 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.401873    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj76d\" (UniqueName: \"kubernetes.io/projected/1e73c307-868e-40e8-ad04-099a7446ccef-kube-api-access-fj76d\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:14.401908 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.401910    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:14.402175 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.401924    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:14.402175 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.401939    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:14.402175 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.401953    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e73c307-868e-40e8-ad04-099a7446ccef-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:14.402175 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.401966    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e73c307-868e-40e8-ad04-099a7446ccef-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:14.402175 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.401978    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e73c307-868e-40e8-ad04-099a7446ccef-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:14.402175 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.401991    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e73c307-868e-40e8-ad04-099a7446ccef-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:14.423476 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.423443    4924 generic.go:356] "Generic (PLEG): container finished" podID="1e73c307-868e-40e8-ad04-099a7446ccef" containerID="3652bb640212f6cdd25b148d97e83be90cf657b4972e7f0ae17815306ad84c62" exitCode=0
Oct 30 19:18:14.423632 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.423496    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"
Oct 30 19:18:14.423632 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.423525    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f" event={"ID":"1e73c307-868e-40e8-ad04-099a7446ccef","Type":"ContainerDied","Data":"3652bb640212f6cdd25b148d97e83be90cf657b4972e7f0ae17815306ad84c62"}
Oct 30 19:18:14.423632 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.423580    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f" event={"ID":"1e73c307-868e-40e8-ad04-099a7446ccef","Type":"ContainerDied","Data":"6498b507b4a3daf039ad4268e7735c6a1f04ebb95a3b17f7e74961e3338c2d32"}
Oct 30 19:18:14.423632 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.423603    4924 scope.go:122] "RemoveContainer" containerID="3652bb640212f6cdd25b148d97e83be90cf657b4972e7f0ae17815306ad84c62"
Oct 30 19:18:14.437423 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.437396    4924 scope.go:122] "RemoveContainer" containerID="a3e84af7e790444c71aebec9462ee8037428558eaad1a91b9c35353b317a238e"
Oct 30 19:18:14.447090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.447054    4924 scope.go:122] "RemoveContainer" containerID="3652bb640212f6cdd25b148d97e83be90cf657b4972e7f0ae17815306ad84c62"
Oct 30 19:18:14.447418 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:18:14.447399    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3652bb640212f6cdd25b148d97e83be90cf657b4972e7f0ae17815306ad84c62\": container with ID starting with 3652bb640212f6cdd25b148d97e83be90cf657b4972e7f0ae17815306ad84c62 not found: ID does not exist" containerID="3652bb640212f6cdd25b148d97e83be90cf657b4972e7f0ae17815306ad84c62"
Oct 30 19:18:14.447485 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.447429    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3652bb640212f6cdd25b148d97e83be90cf657b4972e7f0ae17815306ad84c62"} err="failed to get container status \"3652bb640212f6cdd25b148d97e83be90cf657b4972e7f0ae17815306ad84c62\": rpc error: code = NotFound desc = could not find container \"3652bb640212f6cdd25b148d97e83be90cf657b4972e7f0ae17815306ad84c62\": container with ID starting with 3652bb640212f6cdd25b148d97e83be90cf657b4972e7f0ae17815306ad84c62 not found: ID does not exist"
Oct 30 19:18:14.447485 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.447448    4924 scope.go:122] "RemoveContainer" containerID="a3e84af7e790444c71aebec9462ee8037428558eaad1a91b9c35353b317a238e"
Oct 30 19:18:14.447732 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:18:14.447707    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3e84af7e790444c71aebec9462ee8037428558eaad1a91b9c35353b317a238e\": container with ID starting with a3e84af7e790444c71aebec9462ee8037428558eaad1a91b9c35353b317a238e not found: ID does not exist" containerID="a3e84af7e790444c71aebec9462ee8037428558eaad1a91b9c35353b317a238e"
Oct 30 19:18:14.447785 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.447744    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e84af7e790444c71aebec9462ee8037428558eaad1a91b9c35353b317a238e"} err="failed to get container status \"a3e84af7e790444c71aebec9462ee8037428558eaad1a91b9c35353b317a238e\": rpc error: code = NotFound desc = could not find container \"a3e84af7e790444c71aebec9462ee8037428558eaad1a91b9c35353b317a238e\": container with ID starting with a3e84af7e790444c71aebec9462ee8037428558eaad1a91b9c35353b317a238e not found: ID does not exist"
Oct 30 19:18:14.582168 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.582082    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"]
Oct 30 19:18:14.588346 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.588321    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3b6ad6-openshift-default-654b7b54c9-kzz7f"]
Oct 30 19:18:14.833561 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:14.833481    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e73c307-868e-40e8-ad04-099a7446ccef" path="/var/lib/kubelet/pods/1e73c307-868e-40e8-ad04-099a7446ccef/volumes"
Oct 30 19:18:15.710579 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:18:15.710536    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:18:16.397439 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:16.397406    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:21.194989 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:21.194958    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:21.513416 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:21.513330    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:25.945415 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:25.945366    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:30.717934 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:30.717864    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:35.539826 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:35.539787    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:40.512874 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:40.512837    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:41.209103 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:41.209053    4924 scope.go:122] "RemoveContainer" containerID="47632dd41219d5e9f816453f4807ddfd53f7c227856513a650fa3426cd368f1a"
Oct 30 19:18:41.218562 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:41.218541    4924 scope.go:122] "RemoveContainer" containerID="53e49aa02328855520565f071b92f2eab7b9229cbe331002eb2b75fc98dcf6c4"
Oct 30 19:18:42.381237 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:42.381195    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:45.291681 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:45.291635    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:45.711491 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:18:45.711457    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:18:50.082595 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:50.082558    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:54.916840 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:54.916808    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:18:59.821413 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:18:59.821377    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:04.600434 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:04.600391    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:09.375462 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:09.375398    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:14.267320 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:14.267276    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:15.711078 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:19:15.711026    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:19:19.114324 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:19.114279    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:21.998938 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:21.998858    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b688453-qh27b"]
Oct 30 19:19:21.999339 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:21.999295    4924 cpu_manager.go:414] "containerMap: removing container" podUID="1e73c307-868e-40e8-ad04-099a7446ccef" containerName="istio-proxy"
Oct 30 19:19:21.999339 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:21.999307    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e73c307-868e-40e8-ad04-099a7446ccef" containerName="istio-proxy"
Oct 30 19:19:21.999339 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:21.999333    4924 cpu_manager.go:414] "containerMap: removing container" podUID="539e8daa-2f24-4d43-8017-237fb777d90e" containerName="payload-pre-processing"
Oct 30 19:19:21.999339 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:21.999338    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="539e8daa-2f24-4d43-8017-237fb777d90e" containerName="payload-pre-processing"
Oct 30 19:19:21.999490 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:21.999423    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="539e8daa-2f24-4d43-8017-237fb777d90e" containerName="payload-pre-processing"
Oct 30 19:19:21.999490 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:21.999436    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e73c307-868e-40e8-ad04-099a7446ccef" containerName="istio-proxy"
Oct 30 19:19:21.999490 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:21.999445    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e73c307-868e-40e8-ad04-099a7446ccef" containerName="istio-proxy"
Oct 30 19:19:22.003355 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:22.003332    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b688453-qh27b"
Oct 30 19:19:22.102100 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:22.102047    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:19:22.102293 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:22.102237    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:19:22.102293 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:22.102253    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:19:22.102293 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:22.102267    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:19:22.104962 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:22.104945    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b688453-qh27b"]
Oct 30 19:19:22.126893 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:22.126871    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ee9990-2d90-4e92-b564-121c5080271f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b688453-qh27b\" (UID: \"90ee9990-2d90-4e92-b564-121c5080271f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b688453-qh27b"
Oct 30 19:19:22.127036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:22.126927    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ee9990-2d90-4e92-b564-121c5080271f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b688453-qh27b\" (UID: \"90ee9990-2d90-4e92-b564-121c5080271f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b688453-qh27b"
Oct 30 19:19:22.127036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:22.126950    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq75j\" (UniqueName: \"kubernetes.io/projected/90ee9990-2d90-4e92-b564-121c5080271f-kube-api-access-gq75j\") pod \"maas-api-revoke-sub-7b688453-qh27b\" (UID: \"90ee9990-2d90-4e92-b564-121c5080271f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b688453-qh27b"
Oct 30 19:19:22.228124 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:22.228051    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ee9990-2d90-4e92-b564-121c5080271f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b688453-qh27b\" (UID: \"90ee9990-2d90-4e92-b564-121c5080271f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b688453-qh27b"
Oct 30 19:19:22.228280 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:22.228140    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq75j\" (UniqueName: \"kubernetes.io/projected/90ee9990-2d90-4e92-b564-121c5080271f-kube-api-access-gq75j\") pod \"maas-api-revoke-sub-7b688453-qh27b\" (UID: \"90ee9990-2d90-4e92-b564-121c5080271f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b688453-qh27b"
Oct 30 19:19:22.228326 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:22.228292    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ee9990-2d90-4e92-b564-121c5080271f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b688453-qh27b\" (UID: \"90ee9990-2d90-4e92-b564-121c5080271f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b688453-qh27b"
Oct 30 19:19:22.228747 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:22.228713    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ee9990-2d90-4e92-b564-121c5080271f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b688453-qh27b\" (UID: \"90ee9990-2d90-4e92-b564-121c5080271f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b688453-qh27b"
Oct 30 19:19:22.230635 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:22.230613    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ee9990-2d90-4e92-b564-121c5080271f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b688453-qh27b\" (UID: \"90ee9990-2d90-4e92-b564-121c5080271f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b688453-qh27b"
Oct 30 19:19:22.284949 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:22.284892    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq75j\" (UniqueName: \"kubernetes.io/projected/90ee9990-2d90-4e92-b564-121c5080271f-kube-api-access-gq75j\") pod \"maas-api-revoke-sub-7b688453-qh27b\" (UID: \"90ee9990-2d90-4e92-b564-121c5080271f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b688453-qh27b"
Oct 30 19:19:22.316758 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:22.316731    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b688453-qh27b"
Oct 30 19:19:22.681832 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:22.681801    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b688453-qh27b"]
Oct 30 19:19:22.682926 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:19:22.682896    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ee9990_2d90_4e92_b564_121c5080271f.slice/crio-a969141b4909287279cd03e7df93e3afec0d29faebe098efac8493befddb1236 WatchSource:0}: Error finding container a969141b4909287279cd03e7df93e3afec0d29faebe098efac8493befddb1236: Status 404 returned error can't find the container with id a969141b4909287279cd03e7df93e3afec0d29faebe098efac8493befddb1236
Oct 30 19:19:22.858199 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:22.858164    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b688453-qh27b" event={"ID":"90ee9990-2d90-4e92-b564-121c5080271f","Type":"ContainerStarted","Data":"a969141b4909287279cd03e7df93e3afec0d29faebe098efac8493befddb1236"}
Oct 30 19:19:23.866897 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:23.866807    4924 generic.go:356] "Generic (PLEG): container finished" podID="90ee9990-2d90-4e92-b564-121c5080271f" containerID="999ad7e52f34bfbb51d887f700c2d17cb79e7a1ea55e2e7c9e4ff45b9f898cd9" exitCode=0
Oct 30 19:19:23.867310 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:23.866888    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b688453-qh27b" event={"ID":"90ee9990-2d90-4e92-b564-121c5080271f","Type":"ContainerDied","Data":"999ad7e52f34bfbb51d887f700c2d17cb79e7a1ea55e2e7c9e4ff45b9f898cd9"}
Oct 30 19:19:23.984708 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:23.984675    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:25.014297 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:25.014274    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b688453-qh27b"
Oct 30 19:19:25.160570 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:25.160482    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ee9990-2d90-4e92-b564-121c5080271f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ee9990-2d90-4e92-b564-121c5080271f-maas-api-openshift-ca\") pod \"90ee9990-2d90-4e92-b564-121c5080271f\" (UID: \"90ee9990-2d90-4e92-b564-121c5080271f\") "
Oct 30 19:19:25.160754 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:25.160590    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ee9990-2d90-4e92-b564-121c5080271f-kube-api-access-gq75j\" (UniqueName: \"kubernetes.io/projected/90ee9990-2d90-4e92-b564-121c5080271f-kube-api-access-gq75j\") pod \"90ee9990-2d90-4e92-b564-121c5080271f\" (UID: \"90ee9990-2d90-4e92-b564-121c5080271f\") "
Oct 30 19:19:25.160754 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:25.160678    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ee9990-2d90-4e92-b564-121c5080271f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ee9990-2d90-4e92-b564-121c5080271f-maas-api-xks-ca\") pod \"90ee9990-2d90-4e92-b564-121c5080271f\" (UID: \"90ee9990-2d90-4e92-b564-121c5080271f\") "
Oct 30 19:19:25.160932 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:25.160895    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ee9990-2d90-4e92-b564-121c5080271f-maas-api-openshift-ca" pod "90ee9990-2d90-4e92-b564-121c5080271f" (UID: "90ee9990-2d90-4e92-b564-121c5080271f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:25.161113 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:25.161059    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ee9990-2d90-4e92-b564-121c5080271f-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:25.162906 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:25.162884    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ee9990-2d90-4e92-b564-121c5080271f-kube-api-access-gq75j" pod "90ee9990-2d90-4e92-b564-121c5080271f" (UID: "90ee9990-2d90-4e92-b564-121c5080271f"). InnerVolumeSpecName "kube-api-access-gq75j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:25.163105 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:25.163074    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ee9990-2d90-4e92-b564-121c5080271f-maas-api-xks-ca" pod "90ee9990-2d90-4e92-b564-121c5080271f" (UID: "90ee9990-2d90-4e92-b564-121c5080271f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:19:25.262436 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:25.262404    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq75j\" (UniqueName: \"kubernetes.io/projected/90ee9990-2d90-4e92-b564-121c5080271f-kube-api-access-gq75j\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:25.262436 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:25.262434    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ee9990-2d90-4e92-b564-121c5080271f-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:25.882137 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:25.882103    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b688453-qh27b" event={"ID":"90ee9990-2d90-4e92-b564-121c5080271f","Type":"ContainerDied","Data":"a969141b4909287279cd03e7df93e3afec0d29faebe098efac8493befddb1236"}
Oct 30 19:19:25.882312 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:25.882154    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a969141b4909287279cd03e7df93e3afec0d29faebe098efac8493befddb1236"
Oct 30 19:19:25.882312 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:25.882122    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b688453-qh27b"
Oct 30 19:19:28.752772 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:28.752738    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:33.506340 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:33.506302    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:33.954192 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:33.954153    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"]
Oct 30 19:19:33.974405 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:33.974363    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:34.892586 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:34.892547    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb02e581-v9cws"]
Oct 30 19:19:34.893206 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:34.893184    4924 cpu_manager.go:414] "containerMap: removing container" podUID="90ee9990-2d90-4e92-b564-121c5080271f" containerName="revoke-keys"
Oct 30 19:19:34.893313 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:34.893206    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ee9990-2d90-4e92-b564-121c5080271f" containerName="revoke-keys"
Oct 30 19:19:34.893313 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:34.893252    4924 cpu_manager.go:414] "containerMap: removing container" podUID="1e73c307-868e-40e8-ad04-099a7446ccef" containerName="istio-proxy"
Oct 30 19:19:34.893313 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:34.893261    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e73c307-868e-40e8-ad04-099a7446ccef" containerName="istio-proxy"
Oct 30 19:19:34.893489 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:34.893431    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ee9990-2d90-4e92-b564-121c5080271f" containerName="revoke-keys"
Oct 30 19:19:34.897103 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:34.897081    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb02e581-v9cws"
Oct 30 19:19:34.951187 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:34.951159    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb02e581-v9cws"]
Oct 30 19:19:34.954247 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:34.954221    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd87q\" (UniqueName: \"kubernetes.io/projected/a242e375-9a2f-4501-bbd4-4f031a3d46f9-kube-api-access-zd87q\") pod \"maas-api-revoke-sub-fb02e581-v9cws\" (UID: \"a242e375-9a2f-4501-bbd4-4f031a3d46f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb02e581-v9cws"
Oct 30 19:19:34.954383 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:34.954264    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a242e375-9a2f-4501-bbd4-4f031a3d46f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb02e581-v9cws\" (UID: \"a242e375-9a2f-4501-bbd4-4f031a3d46f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb02e581-v9cws"
Oct 30 19:19:34.954383 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:34.954365    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a242e375-9a2f-4501-bbd4-4f031a3d46f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb02e581-v9cws\" (UID: \"a242e375-9a2f-4501-bbd4-4f031a3d46f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb02e581-v9cws"
Oct 30 19:19:34.994962 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:34.994933    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:19:34.995128 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:34.994967    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:19:34.995128 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:34.995002    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:19:34.995235 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:34.995004    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:19:35.055598 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:35.055568    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a242e375-9a2f-4501-bbd4-4f031a3d46f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb02e581-v9cws\" (UID: \"a242e375-9a2f-4501-bbd4-4f031a3d46f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb02e581-v9cws"
Oct 30 19:19:35.055738 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:35.055667    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd87q\" (UniqueName: \"kubernetes.io/projected/a242e375-9a2f-4501-bbd4-4f031a3d46f9-kube-api-access-zd87q\") pod \"maas-api-revoke-sub-fb02e581-v9cws\" (UID: \"a242e375-9a2f-4501-bbd4-4f031a3d46f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb02e581-v9cws"
Oct 30 19:19:35.055738 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:35.055699    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a242e375-9a2f-4501-bbd4-4f031a3d46f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb02e581-v9cws\" (UID: \"a242e375-9a2f-4501-bbd4-4f031a3d46f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb02e581-v9cws"
Oct 30 19:19:35.056327 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:35.056306    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a242e375-9a2f-4501-bbd4-4f031a3d46f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb02e581-v9cws\" (UID: \"a242e375-9a2f-4501-bbd4-4f031a3d46f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb02e581-v9cws"
Oct 30 19:19:35.057964 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:35.057939    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a242e375-9a2f-4501-bbd4-4f031a3d46f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb02e581-v9cws\" (UID: \"a242e375-9a2f-4501-bbd4-4f031a3d46f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb02e581-v9cws"
Oct 30 19:19:35.109737 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:35.109705    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd87q\" (UniqueName: \"kubernetes.io/projected/a242e375-9a2f-4501-bbd4-4f031a3d46f9-kube-api-access-zd87q\") pod \"maas-api-revoke-sub-fb02e581-v9cws\" (UID: \"a242e375-9a2f-4501-bbd4-4f031a3d46f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb02e581-v9cws"
Oct 30 19:19:35.210136 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:35.210019    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb02e581-v9cws"
Oct 30 19:19:35.576280 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:35.576253    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb02e581-v9cws"]
Oct 30 19:19:35.583722 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:19:35.583687    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda242e375_9a2f_4501_bbd4_4f031a3d46f9.slice/crio-9f13dcdf8d4c957fde43b2df44369e07103a6b83c74ef15002030ec003835ded WatchSource:0}: Error finding container 9f13dcdf8d4c957fde43b2df44369e07103a6b83c74ef15002030ec003835ded: Status 404 returned error can't find the container with id 9f13dcdf8d4c957fde43b2df44369e07103a6b83c74ef15002030ec003835ded
Oct 30 19:19:35.947147 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:35.947110    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb02e581-v9cws" event={"ID":"a242e375-9a2f-4501-bbd4-4f031a3d46f9","Type":"ContainerStarted","Data":"4f68f0b9d5840f140843d4e96922a9f2916d21c2c81217ca22334d7d83a9269f"}
Oct 30 19:19:35.947560 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:35.947171    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb02e581-v9cws" event={"ID":"a242e375-9a2f-4501-bbd4-4f031a3d46f9","Type":"ContainerStarted","Data":"9f13dcdf8d4c957fde43b2df44369e07103a6b83c74ef15002030ec003835ded"}
Oct 30 19:19:36.078771 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:36.078714    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb02e581-v9cws" podStartSLOduration=2.078696039 podStartE2EDuration="2.078696039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:36.0757519 +0000 UTC m=+2577.352802345" watchObservedRunningTime="2026-10-30 19:19:36.078696039 +0000 UTC m=+2577.355746485"
Oct 30 19:19:36.954957 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:36.954925    4924 generic.go:356] "Generic (PLEG): container finished" podID="a242e375-9a2f-4501-bbd4-4f031a3d46f9" containerID="4f68f0b9d5840f140843d4e96922a9f2916d21c2c81217ca22334d7d83a9269f" exitCode=0
Oct 30 19:19:36.955382 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:36.955010    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb02e581-v9cws" event={"ID":"a242e375-9a2f-4501-bbd4-4f031a3d46f9","Type":"ContainerDied","Data":"4f68f0b9d5840f140843d4e96922a9f2916d21c2c81217ca22334d7d83a9269f"}
Oct 30 19:19:38.120461 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:38.120432    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb02e581-v9cws"
Oct 30 19:19:38.186338 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:38.186308    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a242e375-9a2f-4501-bbd4-4f031a3d46f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a242e375-9a2f-4501-bbd4-4f031a3d46f9-maas-api-xks-ca\") pod \"a242e375-9a2f-4501-bbd4-4f031a3d46f9\" (UID: \"a242e375-9a2f-4501-bbd4-4f031a3d46f9\") "
Oct 30 19:19:38.186494 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:38.186426    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a242e375-9a2f-4501-bbd4-4f031a3d46f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a242e375-9a2f-4501-bbd4-4f031a3d46f9-maas-api-openshift-ca\") pod \"a242e375-9a2f-4501-bbd4-4f031a3d46f9\" (UID: \"a242e375-9a2f-4501-bbd4-4f031a3d46f9\") "
Oct 30 19:19:38.186552 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:38.186536    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a242e375-9a2f-4501-bbd4-4f031a3d46f9-kube-api-access-zd87q\" (UniqueName: \"kubernetes.io/projected/a242e375-9a2f-4501-bbd4-4f031a3d46f9-kube-api-access-zd87q\") pod \"a242e375-9a2f-4501-bbd4-4f031a3d46f9\" (UID: \"a242e375-9a2f-4501-bbd4-4f031a3d46f9\") "
Oct 30 19:19:38.186848 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:38.186823    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a242e375-9a2f-4501-bbd4-4f031a3d46f9-maas-api-openshift-ca" pod "a242e375-9a2f-4501-bbd4-4f031a3d46f9" (UID: "a242e375-9a2f-4501-bbd4-4f031a3d46f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:38.188366 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:38.188341    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a242e375-9a2f-4501-bbd4-4f031a3d46f9-maas-api-xks-ca" pod "a242e375-9a2f-4501-bbd4-4f031a3d46f9" (UID: "a242e375-9a2f-4501-bbd4-4f031a3d46f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:19:38.188467 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:38.188413    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a242e375-9a2f-4501-bbd4-4f031a3d46f9-kube-api-access-zd87q" pod "a242e375-9a2f-4501-bbd4-4f031a3d46f9" (UID: "a242e375-9a2f-4501-bbd4-4f031a3d46f9"). InnerVolumeSpecName "kube-api-access-zd87q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:38.288190 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:38.288110    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd87q\" (UniqueName: \"kubernetes.io/projected/a242e375-9a2f-4501-bbd4-4f031a3d46f9-kube-api-access-zd87q\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:38.288190 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:38.288140    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a242e375-9a2f-4501-bbd4-4f031a3d46f9-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:38.288190 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:38.288150    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a242e375-9a2f-4501-bbd4-4f031a3d46f9-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:38.455117 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:38.455086    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:38.968038 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:38.968011    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb02e581-v9cws"
Oct 30 19:19:38.968224 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:38.968053    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb02e581-v9cws" event={"ID":"a242e375-9a2f-4501-bbd4-4f031a3d46f9","Type":"ContainerDied","Data":"9f13dcdf8d4c957fde43b2df44369e07103a6b83c74ef15002030ec003835ded"}
Oct 30 19:19:38.968224 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:38.968122    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f13dcdf8d4c957fde43b2df44369e07103a6b83c74ef15002030ec003835ded"
Oct 30 19:19:41.276389 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:41.276349    4924 scope.go:122] "RemoveContainer" containerID="5adc89a054050e72631c3ff58f1a7c7588ce0b9eb03ad3a125551c54604a7ec5"
Oct 30 19:19:41.286263 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:41.286242    4924 scope.go:122] "RemoveContainer" containerID="30c6d97850af0a5860ed64fa0f0a597db1835dbbc06e3ef1570150eafc0985ba"
Oct 30 19:19:41.294949 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:41.294931    4924 scope.go:122] "RemoveContainer" containerID="101cde382f9a3c71f8b44f7c7635c4f7bde7e900cf26c9f298d1810d898b6d59"
Oct 30 19:19:42.707052 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:42.707020    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:43.191660 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:43.191628    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:45.711095 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:19:45.711036    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:19:48.092802 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:48.092767    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:48.973928 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:48.973891    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5" podUID="9bbf86bc-d98c-4c76-b96e-ff9a48795800" containerName="main" containerID="cri-o://0a0ca4b06f021572fe567e03033e89e7c78f981c6243d6d808f2d5f97f404ee9" gracePeriod=45
Oct 30 19:19:49.014894 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:19:49.014865    4924 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bbf86bc_d98c_4c76_b96e_ff9a48795800.slice/crio-0a0ca4b06f021572fe567e03033e89e7c78f981c6243d6d808f2d5f97f404ee9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bbf86bc_d98c_4c76_b96e_ff9a48795800.slice/crio-conmon-0a0ca4b06f021572fe567e03033e89e7c78f981c6243d6d808f2d5f97f404ee9.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:19:49.235745 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:49.235683    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"
Oct 30 19:19:49.306394 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:49.306363    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bbf86bc-d98c-4c76-b96e-ff9a48795800-kube-api-access-bcdnw\" (UniqueName: \"kubernetes.io/projected/9bbf86bc-d98c-4c76-b96e-ff9a48795800-kube-api-access-bcdnw\") pod \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\" (UID: \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\") "
Oct 30 19:19:49.306538 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:49.306409    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-tmp-dir\") pod \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\" (UID: \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\") "
Oct 30 19:19:49.306538 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:49.306509    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-home\" (UniqueName: \"kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-home\") pod \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\" (UID: \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\") "
Oct 30 19:19:49.306653 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:49.306607    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bbf86bc-d98c-4c76-b96e-ff9a48795800-tls-certs\" (UniqueName: \"kubernetes.io/secret/9bbf86bc-d98c-4c76-b96e-ff9a48795800-tls-certs\") pod \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\" (UID: \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\") "
Oct 30 19:19:49.306712 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:49.306649    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-tmp-dir" pod "9bbf86bc-d98c-4c76-b96e-ff9a48795800" (UID: "9bbf86bc-d98c-4c76-b96e-ff9a48795800"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:49.306774 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:49.306726    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-dshm\") pod \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\" (UID: \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\") "
Oct 30 19:19:49.306774 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:49.306738    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-home" pod "9bbf86bc-d98c-4c76-b96e-ff9a48795800" (UID: "9bbf86bc-d98c-4c76-b96e-ff9a48795800"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:49.306774 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:49.306762    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-model-cache\") pod \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\" (UID: \"9bbf86bc-d98c-4c76-b96e-ff9a48795800\") "
Oct 30 19:19:49.307056 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:49.307030    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-model-cache" pod "9bbf86bc-d98c-4c76-b96e-ff9a48795800" (UID: "9bbf86bc-d98c-4c76-b96e-ff9a48795800"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:49.307188 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:49.307173    4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-model-cache\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:49.307257 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:49.307191    4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-tmp-dir\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:49.307257 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:49.307200    4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-home\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:49.308747 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:49.308720    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bbf86bc-d98c-4c76-b96e-ff9a48795800-tls-certs" pod "9bbf86bc-d98c-4c76-b96e-ff9a48795800" (UID: "9bbf86bc-d98c-4c76-b96e-ff9a48795800"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:19:49.308849 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:49.308771    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-dshm" pod "9bbf86bc-d98c-4c76-b96e-ff9a48795800" (UID: "9bbf86bc-d98c-4c76-b96e-ff9a48795800"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:19:49.308849 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:49.308792    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bbf86bc-d98c-4c76-b96e-ff9a48795800-kube-api-access-bcdnw" pod "9bbf86bc-d98c-4c76-b96e-ff9a48795800" (UID: "9bbf86bc-d98c-4c76-b96e-ff9a48795800"). InnerVolumeSpecName "kube-api-access-bcdnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:49.408569 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:49.408532    4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bbf86bc-d98c-4c76-b96e-ff9a48795800-tls-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:49.408569 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:49.408564    4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bbf86bc-d98c-4c76-b96e-ff9a48795800-dshm\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:49.408569 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:49.408574    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcdnw\" (UniqueName: \"kubernetes.io/projected/9bbf86bc-d98c-4c76-b96e-ff9a48795800-kube-api-access-bcdnw\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:50.045286 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.045249    4924 generic.go:356] "Generic (PLEG): container finished" podID="9bbf86bc-d98c-4c76-b96e-ff9a48795800" containerID="0a0ca4b06f021572fe567e03033e89e7c78f981c6243d6d808f2d5f97f404ee9" exitCode=0
Oct 30 19:19:50.045495 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.045302    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"
Oct 30 19:19:50.045495 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.045327    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5" event={"ID":"9bbf86bc-d98c-4c76-b96e-ff9a48795800","Type":"ContainerDied","Data":"0a0ca4b06f021572fe567e03033e89e7c78f981c6243d6d808f2d5f97f404ee9"}
Oct 30 19:19:50.045495 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.045373    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5" event={"ID":"9bbf86bc-d98c-4c76-b96e-ff9a48795800","Type":"ContainerDied","Data":"68354f4da31b4784ad198138a77a0f0f985d26e9cdcbf4fca981004930cfdfa8"}
Oct 30 19:19:50.045495 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.045389    4924 scope.go:122] "RemoveContainer" containerID="0a0ca4b06f021572fe567e03033e89e7c78f981c6243d6d808f2d5f97f404ee9"
Oct 30 19:19:50.055361 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.055340    4924 scope.go:122] "RemoveContainer" containerID="0a0ca4b06f021572fe567e03033e89e7c78f981c6243d6d808f2d5f97f404ee9"
Oct 30 19:19:50.055630 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:19:50.055607    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a0ca4b06f021572fe567e03033e89e7c78f981c6243d6d808f2d5f97f404ee9\": container with ID starting with 0a0ca4b06f021572fe567e03033e89e7c78f981c6243d6d808f2d5f97f404ee9 not found: ID does not exist" containerID="0a0ca4b06f021572fe567e03033e89e7c78f981c6243d6d808f2d5f97f404ee9"
Oct 30 19:19:50.055694 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.055640    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a0ca4b06f021572fe567e03033e89e7c78f981c6243d6d808f2d5f97f404ee9"} err="failed to get container status \"0a0ca4b06f021572fe567e03033e89e7c78f981c6243d6d808f2d5f97f404ee9\": rpc error: code = NotFound desc = could not find container \"0a0ca4b06f021572fe567e03033e89e7c78f981c6243d6d808f2d5f97f404ee9\": container with ID starting with 0a0ca4b06f021572fe567e03033e89e7c78f981c6243d6d808f2d5f97f404ee9 not found: ID does not exist"
Oct 30 19:19:50.201712 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.201677    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"]
Oct 30 19:19:50.248399 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.248364    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f75d5f/test-model-f75d5f-kserve-7767fbbdf-4fmk5"]
Oct 30 19:19:50.303141 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.303047    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"]
Oct 30 19:19:50.365398 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.365369    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:50.623146 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.623115    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr"]
Oct 30 19:19:50.623559 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.623545    4924 cpu_manager.go:414] "containerMap: removing container" podUID="9bbf86bc-d98c-4c76-b96e-ff9a48795800" containerName="main"
Oct 30 19:19:50.623559 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.623561    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bbf86bc-d98c-4c76-b96e-ff9a48795800" containerName="main"
Oct 30 19:19:50.623659 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.623591    4924 cpu_manager.go:414] "containerMap: removing container" podUID="a242e375-9a2f-4501-bbd4-4f031a3d46f9" containerName="revoke-keys"
Oct 30 19:19:50.623659 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.623597    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a242e375-9a2f-4501-bbd4-4f031a3d46f9" containerName="revoke-keys"
Oct 30 19:19:50.623731 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.623672    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bbf86bc-d98c-4c76-b96e-ff9a48795800" containerName="main"
Oct 30 19:19:50.623731 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.623685    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="a242e375-9a2f-4501-bbd4-4f031a3d46f9" containerName="revoke-keys"
Oct 30 19:19:50.628397 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.628378    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr"
Oct 30 19:19:50.678265 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.678234    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:19:50.678265 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.678249    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:19:50.679044 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.679018    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:19:50.679191 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.679029    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:19:50.681313 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.681291    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr"]
Oct 30 19:19:50.722724 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.722686    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cad3b97-e083-4472-8be3-21ccea98912d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr\" (UID: \"6cad3b97-e083-4472-8be3-21ccea98912d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr"
Oct 30 19:19:50.722896 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.722753    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk8fp\" (UniqueName: \"kubernetes.io/projected/6cad3b97-e083-4472-8be3-21ccea98912d-kube-api-access-rk8fp\") pod \"maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr\" (UID: \"6cad3b97-e083-4472-8be3-21ccea98912d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr"
Oct 30 19:19:50.722896 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.722833    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cad3b97-e083-4472-8be3-21ccea98912d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr\" (UID: \"6cad3b97-e083-4472-8be3-21ccea98912d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr"
Oct 30 19:19:50.824257 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.824220    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cad3b97-e083-4472-8be3-21ccea98912d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr\" (UID: \"6cad3b97-e083-4472-8be3-21ccea98912d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr"
Oct 30 19:19:50.824451 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.824294    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk8fp\" (UniqueName: \"kubernetes.io/projected/6cad3b97-e083-4472-8be3-21ccea98912d-kube-api-access-rk8fp\") pod \"maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr\" (UID: \"6cad3b97-e083-4472-8be3-21ccea98912d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr"
Oct 30 19:19:50.824451 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.824321    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cad3b97-e083-4472-8be3-21ccea98912d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr\" (UID: \"6cad3b97-e083-4472-8be3-21ccea98912d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr"
Oct 30 19:19:50.824987 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.824960    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cad3b97-e083-4472-8be3-21ccea98912d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr\" (UID: \"6cad3b97-e083-4472-8be3-21ccea98912d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr"
Oct 30 19:19:50.826649 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.826624    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cad3b97-e083-4472-8be3-21ccea98912d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr\" (UID: \"6cad3b97-e083-4472-8be3-21ccea98912d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr"
Oct 30 19:19:50.835122 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.835094    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bbf86bc-d98c-4c76-b96e-ff9a48795800" path="/var/lib/kubelet/pods/9bbf86bc-d98c-4c76-b96e-ff9a48795800/volumes"
Oct 30 19:19:50.882606 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.882547    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk8fp\" (UniqueName: \"kubernetes.io/projected/6cad3b97-e083-4472-8be3-21ccea98912d-kube-api-access-rk8fp\") pod \"maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr\" (UID: \"6cad3b97-e083-4472-8be3-21ccea98912d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr"
Oct 30 19:19:50.940931 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:50.940903    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr"
Oct 30 19:19:51.312955 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:51.312826    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr"]
Oct 30 19:19:52.059499 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:52.059408    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr" event={"ID":"6cad3b97-e083-4472-8be3-21ccea98912d","Type":"ContainerStarted","Data":"82e42c2f6a1e2e117f5d92fd07e6589b8d649c71c28d310c24de75492eef80ef"}
Oct 30 19:19:52.059499 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:52.059464    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr" event={"ID":"6cad3b97-e083-4472-8be3-21ccea98912d","Type":"ContainerStarted","Data":"9210ca4dbcfee289f2c5cd99fb61ab9aeae594ade247e08acd1fa681a9c72d0c"}
Oct 30 19:19:52.168855 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:52.168803    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr" podStartSLOduration=2.168787565 podStartE2EDuration="2.168787565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:52.16604305 +0000 UTC m=+2593.443093495" watchObservedRunningTime="2026-10-30 19:19:52.168787565 +0000 UTC m=+2593.445838054"
Oct 30 19:19:53.067319 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:53.067282    4924 generic.go:356] "Generic (PLEG): container finished" podID="6cad3b97-e083-4472-8be3-21ccea98912d" containerID="82e42c2f6a1e2e117f5d92fd07e6589b8d649c71c28d310c24de75492eef80ef" exitCode=0
Oct 30 19:19:53.067713 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:53.067338    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr" event={"ID":"6cad3b97-e083-4472-8be3-21ccea98912d","Type":"ContainerDied","Data":"82e42c2f6a1e2e117f5d92fd07e6589b8d649c71c28d310c24de75492eef80ef"}
Oct 30 19:19:53.164080 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:53.164032    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:54.216216 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:54.216192    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr"
Oct 30 19:19:54.360614 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:54.360577    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cad3b97-e083-4472-8be3-21ccea98912d-kube-api-access-rk8fp\" (UniqueName: \"kubernetes.io/projected/6cad3b97-e083-4472-8be3-21ccea98912d-kube-api-access-rk8fp\") pod \"6cad3b97-e083-4472-8be3-21ccea98912d\" (UID: \"6cad3b97-e083-4472-8be3-21ccea98912d\") "
Oct 30 19:19:54.360791 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:54.360714    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cad3b97-e083-4472-8be3-21ccea98912d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cad3b97-e083-4472-8be3-21ccea98912d-maas-api-xks-ca\") pod \"6cad3b97-e083-4472-8be3-21ccea98912d\" (UID: \"6cad3b97-e083-4472-8be3-21ccea98912d\") "
Oct 30 19:19:54.360850 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:54.360834    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cad3b97-e083-4472-8be3-21ccea98912d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cad3b97-e083-4472-8be3-21ccea98912d-maas-api-openshift-ca\") pod \"6cad3b97-e083-4472-8be3-21ccea98912d\" (UID: \"6cad3b97-e083-4472-8be3-21ccea98912d\") "
Oct 30 19:19:54.361226 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:54.361197    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cad3b97-e083-4472-8be3-21ccea98912d-maas-api-openshift-ca" pod "6cad3b97-e083-4472-8be3-21ccea98912d" (UID: "6cad3b97-e083-4472-8be3-21ccea98912d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:19:54.361366 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:54.361296    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cad3b97-e083-4472-8be3-21ccea98912d-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:54.362712 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:54.362684    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cad3b97-e083-4472-8be3-21ccea98912d-kube-api-access-rk8fp" pod "6cad3b97-e083-4472-8be3-21ccea98912d" (UID: "6cad3b97-e083-4472-8be3-21ccea98912d"). InnerVolumeSpecName "kube-api-access-rk8fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:19:54.362804 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:54.362782    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cad3b97-e083-4472-8be3-21ccea98912d-maas-api-xks-ca" pod "6cad3b97-e083-4472-8be3-21ccea98912d" (UID: "6cad3b97-e083-4472-8be3-21ccea98912d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:19:54.462036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:54.461998    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk8fp\" (UniqueName: \"kubernetes.io/projected/6cad3b97-e083-4472-8be3-21ccea98912d-kube-api-access-rk8fp\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:54.462036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:54.462032    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cad3b97-e083-4472-8be3-21ccea98912d-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:19:55.082433 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:55.082400    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr"
Oct 30 19:19:55.082610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:55.082401    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr" event={"ID":"6cad3b97-e083-4472-8be3-21ccea98912d","Type":"ContainerDied","Data":"9210ca4dbcfee289f2c5cd99fb61ab9aeae594ade247e08acd1fa681a9c72d0c"}
Oct 30 19:19:55.082610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:55.082526    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9210ca4dbcfee289f2c5cd99fb61ab9aeae594ade247e08acd1fa681a9c72d0c"
Oct 30 19:19:57.875357 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:57.875322    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:19:58.321778 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:19:58.321690    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:02.005337 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:02.005296    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd"]
Oct 30 19:20:02.370542 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:02.370510    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:02.669551 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:02.669457    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:03.291532 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:03.291498    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:05.323031 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:05.322987    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw" podUID="ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f" containerName="main" containerID="cri-o://066df397d77cff5c3b219cbd8fda2f0ab33d6a18972d13daaa5925a17563f626" gracePeriod=45
Oct 30 19:20:05.574384 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:05.574322    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"
Oct 30 19:20:05.684308 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:05.684274    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-tls-certs\") pod \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\" (UID: \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\") "
Oct 30 19:20:05.684501 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:05.684318    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-tmp-dir\") pod \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\" (UID: \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\") "
Oct 30 19:20:05.684501 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:05.684350    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-dshm\") pod \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\" (UID: \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\") "
Oct 30 19:20:05.684501 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:05.684449    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-home\" (UniqueName: \"kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-home\") pod \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\" (UID: \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\") "
Oct 30 19:20:05.684501 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:05.684484    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-kube-api-access-5td4r\" (UniqueName: \"kubernetes.io/projected/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-kube-api-access-5td4r\") pod \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\" (UID: \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\") "
Oct 30 19:20:05.684757 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:05.684516    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-model-cache\") pod \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\" (UID: \"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f\") "
Oct 30 19:20:05.684757 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:05.684664    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-tmp-dir" pod "ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f" (UID: "ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:05.684876 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:05.684827    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-home" pod "ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f" (UID: "ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:05.685147 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:05.685049    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-model-cache" pod "ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f" (UID: "ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:05.685147 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:05.685097    4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-home\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:05.685147 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:05.685120    4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-tmp-dir\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:05.687282 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:05.687223    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-tls-certs" pod "ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f" (UID: "ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:05.687282 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:05.687233    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-dshm" pod "ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f" (UID: "ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:05.687719 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:05.687693    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-kube-api-access-5td4r" pod "ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f" (UID: "ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f"). InnerVolumeSpecName "kube-api-access-5td4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:05.785675 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:05.785638    4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-tls-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:05.785675 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:05.785669    4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-dshm\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:05.785675 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:05.785678    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5td4r\" (UniqueName: \"kubernetes.io/projected/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-kube-api-access-5td4r\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:05.785925 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:05.785689    4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f-model-cache\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:06.151076 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:06.151040    4924 generic.go:356] "Generic (PLEG): container finished" podID="ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f" containerID="066df397d77cff5c3b219cbd8fda2f0ab33d6a18972d13daaa5925a17563f626" exitCode=0
Oct 30 19:20:06.151261 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:06.151108    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"
Oct 30 19:20:06.151261 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:06.151111    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw" event={"ID":"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f","Type":"ContainerDied","Data":"066df397d77cff5c3b219cbd8fda2f0ab33d6a18972d13daaa5925a17563f626"}
Oct 30 19:20:06.151261 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:06.151158    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw" event={"ID":"ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f","Type":"ContainerDied","Data":"c4adfb30159ef100cbdb6ec6b7c04d413777a974a6633d4f6919fe22cc5bafbe"}
Oct 30 19:20:06.151261 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:06.151175    4924 scope.go:122] "RemoveContainer" containerID="066df397d77cff5c3b219cbd8fda2f0ab33d6a18972d13daaa5925a17563f626"
Oct 30 19:20:06.162392 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:06.162370    4924 scope.go:122] "RemoveContainer" containerID="066df397d77cff5c3b219cbd8fda2f0ab33d6a18972d13daaa5925a17563f626"
Oct 30 19:20:06.162683 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:20:06.162663    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"066df397d77cff5c3b219cbd8fda2f0ab33d6a18972d13daaa5925a17563f626\": container with ID starting with 066df397d77cff5c3b219cbd8fda2f0ab33d6a18972d13daaa5925a17563f626 not found: ID does not exist" containerID="066df397d77cff5c3b219cbd8fda2f0ab33d6a18972d13daaa5925a17563f626"
Oct 30 19:20:06.162739 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:06.162693    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"066df397d77cff5c3b219cbd8fda2f0ab33d6a18972d13daaa5925a17563f626"} err="failed to get container status \"066df397d77cff5c3b219cbd8fda2f0ab33d6a18972d13daaa5925a17563f626\": rpc error: code = NotFound desc = could not find container \"066df397d77cff5c3b219cbd8fda2f0ab33d6a18972d13daaa5925a17563f626\": container with ID starting with 066df397d77cff5c3b219cbd8fda2f0ab33d6a18972d13daaa5925a17563f626 not found: ID does not exist"
Oct 30 19:20:06.314332 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:06.314224    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"]
Oct 30 19:20:06.360946 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:06.360916    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-29de7b/test-model-29de7b-kserve-5798d9fcbf-vssfw"]
Oct 30 19:20:06.834360 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:06.834329    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f" path="/var/lib/kubelet/pods/ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f/volumes"
Oct 30 19:20:07.005774 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:07.005718    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd" podUID="c05d45b0-ed8c-4682-872c-f8ac6e5b907e" containerName="payload-processing" containerID="cri-o://76be321eb639f1f7fff11c177aac30ba8f789ece27857e4a7597f21b2d92fedc" gracePeriod=25
Oct 30 19:20:08.205685 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:08.205647    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:10.554607 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:10.554564    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:10.853709 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:10.853668    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:13.125825 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:13.125791    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:14.494868 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:14.494830    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd" podUID="c05d45b0-ed8c-4682-872c-f8ac6e5b907e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:20:15.712125 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:20:15.712088    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:20:17.539284 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:17.539247    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:19.105358 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:19.105323    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:19.387447 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:19.387340    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:19.714926 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:19.714836    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr"]
Oct 30 19:20:19.761368 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:19.761322    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f75d5f-l75dr"]
Oct 30 19:20:20.182020 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:20.181985    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:20.833342 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:20.833311    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cad3b97-e083-4472-8be3-21ccea98912d" path="/var/lib/kubelet/pods/6cad3b97-e083-4472-8be3-21ccea98912d/volumes"
Oct 30 19:20:24.436368 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:24.436336    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:24.495308 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:24.495274    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd" podUID="c05d45b0-ed8c-4682-872c-f8ac6e5b907e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:20:25.060631 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:25.060592    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66ebbd68-lrwl9"]
Oct 30 19:20:25.107544 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:25.107513    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6d86012-cbvvz"]
Oct 30 19:20:25.109713 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:25.109699    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6d86012-cbvvz"]
Oct 30 19:20:25.111662 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:25.111643    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66ebbd68-lrwl9"]
Oct 30 19:20:26.833680 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:26.833635    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4087501-1956-4f6e-8b87-9223565cce59" path="/var/lib/kubelet/pods/a4087501-1956-4f6e-8b87-9223565cce59/volumes"
Oct 30 19:20:26.834301 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:26.834279    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06" path="/var/lib/kubelet/pods/c82cfd62-a8cb-45a4-b1a9-3c0acc2b1e06/volumes"
Oct 30 19:20:26.948337 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:26.948291    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:27.264591 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.264545    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:27.281701 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.281662    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"]
Oct 30 19:20:27.281903 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.281882    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745" podUID="4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" containerName="istio-proxy" containerID="cri-o://ce8ad97e0b51f7e259c8991ff62a33b5037bc613b4da58d665e6dd0d99a50e2b" gracePeriod=30
Oct 30 19:20:27.283165 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.283145    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd"
Oct 30 19:20:27.284749 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.284713    4924 generic.go:356] "Generic (PLEG): container finished" podID="c05d45b0-ed8c-4682-872c-f8ac6e5b907e" containerID="76be321eb639f1f7fff11c177aac30ba8f789ece27857e4a7597f21b2d92fedc" exitCode=0
Oct 30 19:20:27.284865 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.284756    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd"
Oct 30 19:20:27.284865 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.284825    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd" event={"ID":"c05d45b0-ed8c-4682-872c-f8ac6e5b907e","Type":"ContainerDied","Data":"76be321eb639f1f7fff11c177aac30ba8f789ece27857e4a7597f21b2d92fedc"}
Oct 30 19:20:27.284993 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.284871    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd" event={"ID":"c05d45b0-ed8c-4682-872c-f8ac6e5b907e","Type":"ContainerDied","Data":"cd4898c4bbe1727d3d04468eb642b2557112e80211a69fc6d23a9c06c516ce55"}
Oct 30 19:20:27.284993 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.284903    4924 scope.go:122] "RemoveContainer" containerID="76be321eb639f1f7fff11c177aac30ba8f789ece27857e4a7597f21b2d92fedc"
Oct 30 19:20:27.294636 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.294610    4924 scope.go:122] "RemoveContainer" containerID="76be321eb639f1f7fff11c177aac30ba8f789ece27857e4a7597f21b2d92fedc"
Oct 30 19:20:27.294936 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:20:27.294916    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76be321eb639f1f7fff11c177aac30ba8f789ece27857e4a7597f21b2d92fedc\": container with ID starting with 76be321eb639f1f7fff11c177aac30ba8f789ece27857e4a7597f21b2d92fedc not found: ID does not exist" containerID="76be321eb639f1f7fff11c177aac30ba8f789ece27857e4a7597f21b2d92fedc"
Oct 30 19:20:27.295014 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.294944    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76be321eb639f1f7fff11c177aac30ba8f789ece27857e4a7597f21b2d92fedc"} err="failed to get container status \"76be321eb639f1f7fff11c177aac30ba8f789ece27857e4a7597f21b2d92fedc\": rpc error: code = NotFound desc = could not find container \"76be321eb639f1f7fff11c177aac30ba8f789ece27857e4a7597f21b2d92fedc\": container with ID starting with 76be321eb639f1f7fff11c177aac30ba8f789ece27857e4a7597f21b2d92fedc not found: ID does not exist"
Oct 30 19:20:27.391107 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.391000    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c05d45b0-ed8c-4682-872c-f8ac6e5b907e-kube-api-access-hn94j\" (UniqueName: \"kubernetes.io/projected/c05d45b0-ed8c-4682-872c-f8ac6e5b907e-kube-api-access-hn94j\") pod \"c05d45b0-ed8c-4682-872c-f8ac6e5b907e\" (UID: \"c05d45b0-ed8c-4682-872c-f8ac6e5b907e\") "
Oct 30 19:20:27.391266 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.391164    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c05d45b0-ed8c-4682-872c-f8ac6e5b907e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05d45b0-ed8c-4682-872c-f8ac6e5b907e-plugins-config-volume\") pod \"c05d45b0-ed8c-4682-872c-f8ac6e5b907e\" (UID: \"c05d45b0-ed8c-4682-872c-f8ac6e5b907e\") "
Oct 30 19:20:27.391532 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.391503    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c05d45b0-ed8c-4682-872c-f8ac6e5b907e-plugins-config-volume" pod "c05d45b0-ed8c-4682-872c-f8ac6e5b907e" (UID: "c05d45b0-ed8c-4682-872c-f8ac6e5b907e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:27.393271 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.393245    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c05d45b0-ed8c-4682-872c-f8ac6e5b907e-kube-api-access-hn94j" pod "c05d45b0-ed8c-4682-872c-f8ac6e5b907e" (UID: "c05d45b0-ed8c-4682-872c-f8ac6e5b907e"). InnerVolumeSpecName "kube-api-access-hn94j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:27.492580 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.492536    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn94j\" (UniqueName: \"kubernetes.io/projected/c05d45b0-ed8c-4682-872c-f8ac6e5b907e-kube-api-access-hn94j\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:27.492580 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.492565    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c05d45b0-ed8c-4682-872c-f8ac6e5b907e-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:27.566761 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.566730    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:27.704224 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.704145    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd"]
Oct 30 19:20:27.710760 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.710730    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f75d5f-7dd4fb564b-n9ngd"]
Oct 30 19:20:27.902337 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.902306    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw"]
Oct 30 19:20:27.902745 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.902718    4924 cpu_manager.go:414] "containerMap: removing container" podUID="6cad3b97-e083-4472-8be3-21ccea98912d" containerName="revoke-keys"
Oct 30 19:20:27.902745 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.902731    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cad3b97-e083-4472-8be3-21ccea98912d" containerName="revoke-keys"
Oct 30 19:20:27.902840 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.902747    4924 cpu_manager.go:414] "containerMap: removing container" podUID="ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f" containerName="main"
Oct 30 19:20:27.902840 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.902753    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f" containerName="main"
Oct 30 19:20:27.902840 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.902773    4924 cpu_manager.go:414] "containerMap: removing container" podUID="c05d45b0-ed8c-4682-872c-f8ac6e5b907e" containerName="payload-processing"
Oct 30 19:20:27.902840 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.902778    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c05d45b0-ed8c-4682-872c-f8ac6e5b907e" containerName="payload-processing"
Oct 30 19:20:27.902990 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.902864    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="c05d45b0-ed8c-4682-872c-f8ac6e5b907e" containerName="payload-processing"
Oct 30 19:20:27.902990 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.902878    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cad3b97-e083-4472-8be3-21ccea98912d" containerName="revoke-keys"
Oct 30 19:20:27.902990 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.902885    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad5175c4-eeb1-4f0f-a8c0-12897baa3b4f" containerName="main"
Oct 30 19:20:27.907455 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.907435    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw"
Oct 30 19:20:27.958060 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.957941    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:20:27.958060 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.957966    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:20:27.958060 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.958005    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:20:27.958333 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.958149    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:20:27.961238 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.961212    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw"]
Oct 30 19:20:27.998126 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.998095    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b583ca-6550-4b34-b9fa-c1fbe15f79e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw\" (UID: \"34b583ca-6550-4b34-b9fa-c1fbe15f79e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw"
Oct 30 19:20:27.998259 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.998159    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jsn7\" (UniqueName: \"kubernetes.io/projected/34b583ca-6550-4b34-b9fa-c1fbe15f79e6-kube-api-access-5jsn7\") pod \"maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw\" (UID: \"34b583ca-6550-4b34-b9fa-c1fbe15f79e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw"
Oct 30 19:20:27.998259 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:27.998233    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b583ca-6550-4b34-b9fa-c1fbe15f79e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw\" (UID: \"34b583ca-6550-4b34-b9fa-c1fbe15f79e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw"
Oct 30 19:20:28.099632 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:28.099596    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jsn7\" (UniqueName: \"kubernetes.io/projected/34b583ca-6550-4b34-b9fa-c1fbe15f79e6-kube-api-access-5jsn7\") pod \"maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw\" (UID: \"34b583ca-6550-4b34-b9fa-c1fbe15f79e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw"
Oct 30 19:20:28.099800 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:28.099640    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b583ca-6550-4b34-b9fa-c1fbe15f79e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw\" (UID: \"34b583ca-6550-4b34-b9fa-c1fbe15f79e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw"
Oct 30 19:20:28.099800 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:28.099707    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b583ca-6550-4b34-b9fa-c1fbe15f79e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw\" (UID: \"34b583ca-6550-4b34-b9fa-c1fbe15f79e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw"
Oct 30 19:20:28.100353 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:28.100329    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b583ca-6550-4b34-b9fa-c1fbe15f79e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw\" (UID: \"34b583ca-6550-4b34-b9fa-c1fbe15f79e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw"
Oct 30 19:20:28.102253 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:28.102233    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b583ca-6550-4b34-b9fa-c1fbe15f79e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw\" (UID: \"34b583ca-6550-4b34-b9fa-c1fbe15f79e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw"
Oct 30 19:20:28.155020 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:28.154994    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jsn7\" (UniqueName: \"kubernetes.io/projected/34b583ca-6550-4b34-b9fa-c1fbe15f79e6-kube-api-access-5jsn7\") pod \"maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw\" (UID: \"34b583ca-6550-4b34-b9fa-c1fbe15f79e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw"
Oct 30 19:20:28.220109 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:28.220027    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw"
Oct 30 19:20:28.585434 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:28.585404    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw"]
Oct 30 19:20:28.587145 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:20:28.587102    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b583ca_6550_4b34_b9fa_c1fbe15f79e6.slice/crio-016024497b9e591bc6fae3850ea4ef1b67635e37eda921c4bfa7d3808d9ac3e7 WatchSource:0}: Error finding container 016024497b9e591bc6fae3850ea4ef1b67635e37eda921c4bfa7d3808d9ac3e7: Status 404 returned error can't find the container with id 016024497b9e591bc6fae3850ea4ef1b67635e37eda921c4bfa7d3808d9ac3e7
Oct 30 19:20:28.833787 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:28.833757    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c05d45b0-ed8c-4682-872c-f8ac6e5b907e" path="/var/lib/kubelet/pods/c05d45b0-ed8c-4682-872c-f8ac6e5b907e/volumes"
Oct 30 19:20:29.195174 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:29.195089    4924 patch_prober.go:29] interesting pod/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:20:29.195174 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:29.195150    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745" podUID="4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:20:29.300139 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:29.300095    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw" event={"ID":"34b583ca-6550-4b34-b9fa-c1fbe15f79e6","Type":"ContainerStarted","Data":"12cdb6dfe0d9dda3c707d4a81aebb6392c5d1a5863b429e046b2fbc627b0e00a"}
Oct 30 19:20:29.300300 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:29.300163    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw" event={"ID":"34b583ca-6550-4b34-b9fa-c1fbe15f79e6","Type":"ContainerStarted","Data":"016024497b9e591bc6fae3850ea4ef1b67635e37eda921c4bfa7d3808d9ac3e7"}
Oct 30 19:20:29.458654 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:29.458557    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw" podStartSLOduration=2.458541647 podStartE2EDuration="2.458541647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:29.411105852 +0000 UTC m=+2630.688156296" watchObservedRunningTime="2026-10-30 19:20:29.458541647 +0000 UTC m=+2630.735592098"
Oct 30 19:20:29.523793 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:29.523765    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:30.307879 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:30.307848    4924 generic.go:356] "Generic (PLEG): container finished" podID="34b583ca-6550-4b34-b9fa-c1fbe15f79e6" containerID="12cdb6dfe0d9dda3c707d4a81aebb6392c5d1a5863b429e046b2fbc627b0e00a" exitCode=0
Oct 30 19:20:30.308315 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:30.307942    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw" event={"ID":"34b583ca-6550-4b34-b9fa-c1fbe15f79e6","Type":"ContainerDied","Data":"12cdb6dfe0d9dda3c707d4a81aebb6392c5d1a5863b429e046b2fbc627b0e00a"}
Oct 30 19:20:31.455227 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:31.455201    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw"
Oct 30 19:20:31.531503 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:31.531470    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b583ca-6550-4b34-b9fa-c1fbe15f79e6-kube-api-access-5jsn7\" (UniqueName: \"kubernetes.io/projected/34b583ca-6550-4b34-b9fa-c1fbe15f79e6-kube-api-access-5jsn7\") pod \"34b583ca-6550-4b34-b9fa-c1fbe15f79e6\" (UID: \"34b583ca-6550-4b34-b9fa-c1fbe15f79e6\") "
Oct 30 19:20:31.531679 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:31.531582    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34b583ca-6550-4b34-b9fa-c1fbe15f79e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b583ca-6550-4b34-b9fa-c1fbe15f79e6-maas-api-xks-ca\") pod \"34b583ca-6550-4b34-b9fa-c1fbe15f79e6\" (UID: \"34b583ca-6550-4b34-b9fa-c1fbe15f79e6\") "
Oct 30 19:20:31.531679 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:31.531635    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b583ca-6550-4b34-b9fa-c1fbe15f79e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b583ca-6550-4b34-b9fa-c1fbe15f79e6-maas-api-openshift-ca\") pod \"34b583ca-6550-4b34-b9fa-c1fbe15f79e6\" (UID: \"34b583ca-6550-4b34-b9fa-c1fbe15f79e6\") "
Oct 30 19:20:31.532038 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:31.532014    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b583ca-6550-4b34-b9fa-c1fbe15f79e6-maas-api-openshift-ca" pod "34b583ca-6550-4b34-b9fa-c1fbe15f79e6" (UID: "34b583ca-6550-4b34-b9fa-c1fbe15f79e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:31.533659 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:31.533635    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34b583ca-6550-4b34-b9fa-c1fbe15f79e6-maas-api-xks-ca" pod "34b583ca-6550-4b34-b9fa-c1fbe15f79e6" (UID: "34b583ca-6550-4b34-b9fa-c1fbe15f79e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:31.533773 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:31.533697    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b583ca-6550-4b34-b9fa-c1fbe15f79e6-kube-api-access-5jsn7" pod "34b583ca-6550-4b34-b9fa-c1fbe15f79e6" (UID: "34b583ca-6550-4b34-b9fa-c1fbe15f79e6"). InnerVolumeSpecName "kube-api-access-5jsn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:31.632844 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:31.632809    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jsn7\" (UniqueName: \"kubernetes.io/projected/34b583ca-6550-4b34-b9fa-c1fbe15f79e6-kube-api-access-5jsn7\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.632844 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:31.632839    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b583ca-6550-4b34-b9fa-c1fbe15f79e6-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:31.632844 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:31.632850    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b583ca-6550-4b34-b9fa-c1fbe15f79e6-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:32.321931 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.321879    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw"
Oct 30 19:20:32.322038 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.321882    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw" event={"ID":"34b583ca-6550-4b34-b9fa-c1fbe15f79e6","Type":"ContainerDied","Data":"016024497b9e591bc6fae3850ea4ef1b67635e37eda921c4bfa7d3808d9ac3e7"}
Oct 30 19:20:32.322038 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.322003    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="016024497b9e591bc6fae3850ea4ef1b67635e37eda921c4bfa7d3808d9ac3e7"
Oct 30 19:20:32.554287 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.554257    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:20:32.641786 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.641749    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-workload-certs\") pod \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") "
Oct 30 19:20:32.641786 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.641792    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-data\") pod \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") "
Oct 30 19:20:32.642030 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.641811    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istiod-ca-cert\") pod \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") "
Oct 30 19:20:32.642030 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.641851    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-podinfo\") pod \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") "
Oct 30 19:20:32.642030 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.641882    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-kube-api-access-wrff4\" (UniqueName: \"kubernetes.io/projected/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-kube-api-access-wrff4\") pod \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") "
Oct 30 19:20:32.642030 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.641910    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-token\" (UniqueName: \"kubernetes.io/projected/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-token\") pod \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") "
Oct 30 19:20:32.642030 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.641978    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-envoy\") pod \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") "
Oct 30 19:20:32.642348 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.642049    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-workload-certs" pod "4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" (UID: "4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:32.642348 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.642087    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-credential-socket\") pod \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") "
Oct 30 19:20:32.642348 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.642093    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-data" pod "4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" (UID: "4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:32.642348 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.642175    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-workload-socket\") pod \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\" (UID: \"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421\") "
Oct 30 19:20:32.642589 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.642417    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-credential-socket" pod "4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" (UID: "4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:32.642589 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.642429    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istiod-ca-cert" pod "4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" (UID: "4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:32.642589 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.642508    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-workload-socket" pod "4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" (UID: "4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:32.642756 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.642653    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:32.642756 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.642678    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:32.642756 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.642693    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:32.642756 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.642708    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:32.642756 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.642722    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:32.644296 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.644267    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-kube-api-access-wrff4" pod "4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" (UID: "4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421"). InnerVolumeSpecName "kube-api-access-wrff4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:32.644296 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.644270    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-podinfo" pod "4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" (UID: "4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:20:32.644529 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.644326    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-token" pod "4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" (UID: "4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:32.644529 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.644329    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-envoy" pod "4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" (UID: "4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:32.743349 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.743309    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:32.743349 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.743340    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:32.743349 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.743351    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrff4\" (UniqueName: \"kubernetes.io/projected/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-kube-api-access-wrff4\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:32.743658 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:32.743361    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:33.330820 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:33.330790    4924 generic.go:356] "Generic (PLEG): container finished" podID="4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" containerID="ce8ad97e0b51f7e259c8991ff62a33b5037bc613b4da58d665e6dd0d99a50e2b" exitCode=0
Oct 30 19:20:33.331002 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:33.330842    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"
Oct 30 19:20:33.331002 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:33.330879    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745" event={"ID":"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421","Type":"ContainerDied","Data":"ce8ad97e0b51f7e259c8991ff62a33b5037bc613b4da58d665e6dd0d99a50e2b"}
Oct 30 19:20:33.331002 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:33.330946    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745" event={"ID":"4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421","Type":"ContainerDied","Data":"e0df7bafac97a16096d052f8a6b0866f525c3b9cdea395b4955f3e004e77bbb7"}
Oct 30 19:20:33.331002 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:33.330976    4924 scope.go:122] "RemoveContainer" containerID="ce8ad97e0b51f7e259c8991ff62a33b5037bc613b4da58d665e6dd0d99a50e2b"
Oct 30 19:20:33.343723 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:33.343699    4924 scope.go:122] "RemoveContainer" containerID="0c9a2365f74bd1886974b2b5ae9c42ad508a8e8a50df847095f1382fcdfb7abe"
Oct 30 19:20:33.353107 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:33.353085    4924 scope.go:122] "RemoveContainer" containerID="ce8ad97e0b51f7e259c8991ff62a33b5037bc613b4da58d665e6dd0d99a50e2b"
Oct 30 19:20:33.353377 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:20:33.353355    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce8ad97e0b51f7e259c8991ff62a33b5037bc613b4da58d665e6dd0d99a50e2b\": container with ID starting with ce8ad97e0b51f7e259c8991ff62a33b5037bc613b4da58d665e6dd0d99a50e2b not found: ID does not exist" containerID="ce8ad97e0b51f7e259c8991ff62a33b5037bc613b4da58d665e6dd0d99a50e2b"
Oct 30 19:20:33.353463 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:33.353391    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8ad97e0b51f7e259c8991ff62a33b5037bc613b4da58d665e6dd0d99a50e2b"} err="failed to get container status \"ce8ad97e0b51f7e259c8991ff62a33b5037bc613b4da58d665e6dd0d99a50e2b\": rpc error: code = NotFound desc = could not find container \"ce8ad97e0b51f7e259c8991ff62a33b5037bc613b4da58d665e6dd0d99a50e2b\": container with ID starting with ce8ad97e0b51f7e259c8991ff62a33b5037bc613b4da58d665e6dd0d99a50e2b not found: ID does not exist"
Oct 30 19:20:33.353463 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:33.353418    4924 scope.go:122] "RemoveContainer" containerID="0c9a2365f74bd1886974b2b5ae9c42ad508a8e8a50df847095f1382fcdfb7abe"
Oct 30 19:20:33.353681 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:20:33.353662    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c9a2365f74bd1886974b2b5ae9c42ad508a8e8a50df847095f1382fcdfb7abe\": container with ID starting with 0c9a2365f74bd1886974b2b5ae9c42ad508a8e8a50df847095f1382fcdfb7abe not found: ID does not exist" containerID="0c9a2365f74bd1886974b2b5ae9c42ad508a8e8a50df847095f1382fcdfb7abe"
Oct 30 19:20:33.353738 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:33.353688    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c9a2365f74bd1886974b2b5ae9c42ad508a8e8a50df847095f1382fcdfb7abe"} err="failed to get container status \"0c9a2365f74bd1886974b2b5ae9c42ad508a8e8a50df847095f1382fcdfb7abe\": rpc error: code = NotFound desc = could not find container \"0c9a2365f74bd1886974b2b5ae9c42ad508a8e8a50df847095f1382fcdfb7abe\": container with ID starting with 0c9a2365f74bd1886974b2b5ae9c42ad508a8e8a50df847095f1382fcdfb7abe not found: ID does not exist"
Oct 30 19:20:33.492765 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:33.492726    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"]
Oct 30 19:20:33.499049 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:33.499016    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f75d5f-openshift-default-75dd9c657c-29745"]
Oct 30 19:20:34.701570 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:34.701542    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:34.834385 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:34.834354    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" path="/var/lib/kubelet/pods/4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421/volumes"
Oct 30 19:20:37.059285 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:37.059241    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88fc83c8-dwnqc"]
Oct 30 19:20:37.103114 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:37.103079    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53345e4b-s92r2"]
Oct 30 19:20:37.105229 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:37.105207    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88fc83c8-dwnqc"]
Oct 30 19:20:37.107035 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:37.107011    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53345e4b-s92r2"]
Oct 30 19:20:38.834599 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:38.834568    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34732c18-0104-4161-8560-5f9f32ebc273" path="/var/lib/kubelet/pods/34732c18-0104-4161-8560-5f9f32ebc273/volumes"
Oct 30 19:20:38.834993 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:38.834977    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64f6635c-af07-453a-85dc-4d76e40b70da" path="/var/lib/kubelet/pods/64f6635c-af07-453a-85dc-4d76e40b70da/volumes"
Oct 30 19:20:39.665053 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:39.664977    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:40.128117 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:40.128082    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:40.432090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:40.431980    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:41.327479 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:41.327444    4924 scope.go:122] "RemoveContainer" containerID="f6bfa4ab57a980a2e1cdc13e8cd890ea0f42fed57825dfcde3a9307833301d8b"
Oct 30 19:20:41.339327 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:41.339306    4924 scope.go:122] "RemoveContainer" containerID="ac99f8e41c5232e4f600909278548fd6293b1c1a1573607ccf2539ac94779a6f"
Oct 30 19:20:41.348619 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:41.348594    4924 scope.go:122] "RemoveContainer" containerID="14ebe99a35906ff46b0ad1d48c65760c56576c2a0963d0522e75ca3fe6bb6b44"
Oct 30 19:20:41.357700 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:41.357674    4924 scope.go:122] "RemoveContainer" containerID="7d194b016d0429eea951d4d25a9b618b44963b87e9ed496bf26f2db1a73b6714"
Oct 30 19:20:43.038123 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:43.038055    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:44.536018 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:44.535987    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:45.711041 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:20:45.711007    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:20:48.818430 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:48.818396    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:50.405198 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:50.405108    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:50.738807 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:50.738724    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:50.902180 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:50.902149    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw"]
Oct 30 19:20:50.947852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:50.947811    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-29de7b-7w6bw"]
Oct 30 19:20:51.349901 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:51.349867    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:52.833328 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:52.833296    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b583ca-6550-4b34-b9fa-c1fbe15f79e6" path="/var/lib/kubelet/pods/34b583ca-6550-4b34-b9fa-c1fbe15f79e6/volumes"
Oct 30 19:20:55.817608 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:55.817576    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:58.400401 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:58.400372    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:58.827843 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:58.827750    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:59.285428 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:59.285398    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:59.571711 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:59.571630    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd"]
Oct 30 19:20:59.572100 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:59.572043    4924 cpu_manager.go:414] "containerMap: removing container" podUID="34b583ca-6550-4b34-b9fa-c1fbe15f79e6" containerName="revoke-keys"
Oct 30 19:20:59.572100 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:59.572054    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b583ca-6550-4b34-b9fa-c1fbe15f79e6" containerName="revoke-keys"
Oct 30 19:20:59.572184 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:59.572109    4924 cpu_manager.go:414] "containerMap: removing container" podUID="4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" containerName="istio-proxy"
Oct 30 19:20:59.572184 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:59.572116    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" containerName="istio-proxy"
Oct 30 19:20:59.572184 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:59.572131    4924 cpu_manager.go:414] "containerMap: removing container" podUID="4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" containerName="istio-proxy"
Oct 30 19:20:59.572184 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:59.572137    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" containerName="istio-proxy"
Oct 30 19:20:59.572330 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:59.572209    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" containerName="istio-proxy"
Oct 30 19:20:59.572330 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:59.572222    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b583ca-6550-4b34-b9fa-c1fbe15f79e6" containerName="revoke-keys"
Oct 30 19:20:59.577526 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:59.577502    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd"
Oct 30 19:20:59.616191 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:59.616161    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93ddea4e-9c77-4dae-8e6b-7e108eebac87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd\" (UID: \"93ddea4e-9c77-4dae-8e6b-7e108eebac87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd"
Oct 30 19:20:59.616352 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:59.616200    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l5sv\" (UniqueName: \"kubernetes.io/projected/93ddea4e-9c77-4dae-8e6b-7e108eebac87-kube-api-access-5l5sv\") pod \"maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd\" (UID: \"93ddea4e-9c77-4dae-8e6b-7e108eebac87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd"
Oct 30 19:20:59.616417 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:59.616397    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93ddea4e-9c77-4dae-8e6b-7e108eebac87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd\" (UID: \"93ddea4e-9c77-4dae-8e6b-7e108eebac87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd"
Oct 30 19:20:59.674429 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:59.674399    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:20:59.674644 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:59.674558    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:20:59.674644 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:59.674591    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:20:59.674644 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:59.674626    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:20:59.677191 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:59.677170    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd"]
Oct 30 19:20:59.717511 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:59.717483    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93ddea4e-9c77-4dae-8e6b-7e108eebac87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd\" (UID: \"93ddea4e-9c77-4dae-8e6b-7e108eebac87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd"
Oct 30 19:20:59.717672 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:59.717532    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93ddea4e-9c77-4dae-8e6b-7e108eebac87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd\" (UID: \"93ddea4e-9c77-4dae-8e6b-7e108eebac87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd"
Oct 30 19:20:59.717672 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:59.717571    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l5sv\" (UniqueName: \"kubernetes.io/projected/93ddea4e-9c77-4dae-8e6b-7e108eebac87-kube-api-access-5l5sv\") pod \"maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd\" (UID: \"93ddea4e-9c77-4dae-8e6b-7e108eebac87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd"
Oct 30 19:20:59.718246 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:59.718222    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93ddea4e-9c77-4dae-8e6b-7e108eebac87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd\" (UID: \"93ddea4e-9c77-4dae-8e6b-7e108eebac87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd"
Oct 30 19:20:59.719877 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:59.719853    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93ddea4e-9c77-4dae-8e6b-7e108eebac87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd\" (UID: \"93ddea4e-9c77-4dae-8e6b-7e108eebac87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd"
Oct 30 19:20:59.773657 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:59.773622    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l5sv\" (UniqueName: \"kubernetes.io/projected/93ddea4e-9c77-4dae-8e6b-7e108eebac87-kube-api-access-5l5sv\") pod \"maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd\" (UID: \"93ddea4e-9c77-4dae-8e6b-7e108eebac87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd"
Oct 30 19:20:59.890101 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:20:59.890060    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd"
Oct 30 19:21:00.260224 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:00.259036    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd"]
Oct 30 19:21:00.496559 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:00.496523    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd" event={"ID":"93ddea4e-9c77-4dae-8e6b-7e108eebac87","Type":"ContainerStarted","Data":"2d8e3a9b073e64b9d406eb1e6fe8a634be6583c0756eef90fe0c436600909b9f"}
Oct 30 19:21:01.125562 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:01.125531    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:01.504327 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:01.504293    4924 generic.go:356] "Generic (PLEG): container finished" podID="93ddea4e-9c77-4dae-8e6b-7e108eebac87" containerID="5d1425fd456b1aa0492e0a1975166984763c4bfd511f790bc8a228a6f2177599" exitCode=0
Oct 30 19:21:01.504548 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:01.504381    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd" event={"ID":"93ddea4e-9c77-4dae-8e6b-7e108eebac87","Type":"ContainerDied","Data":"5d1425fd456b1aa0492e0a1975166984763c4bfd511f790bc8a228a6f2177599"}
Oct 30 19:21:02.656620 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:02.656593    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd"
Oct 30 19:21:02.749283 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:02.749246    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93ddea4e-9c77-4dae-8e6b-7e108eebac87-kube-api-access-5l5sv\" (UniqueName: \"kubernetes.io/projected/93ddea4e-9c77-4dae-8e6b-7e108eebac87-kube-api-access-5l5sv\") pod \"93ddea4e-9c77-4dae-8e6b-7e108eebac87\" (UID: \"93ddea4e-9c77-4dae-8e6b-7e108eebac87\") "
Oct 30 19:21:02.749476 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:02.749300    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93ddea4e-9c77-4dae-8e6b-7e108eebac87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93ddea4e-9c77-4dae-8e6b-7e108eebac87-maas-api-xks-ca\") pod \"93ddea4e-9c77-4dae-8e6b-7e108eebac87\" (UID: \"93ddea4e-9c77-4dae-8e6b-7e108eebac87\") "
Oct 30 19:21:02.749476 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:02.749336    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93ddea4e-9c77-4dae-8e6b-7e108eebac87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93ddea4e-9c77-4dae-8e6b-7e108eebac87-maas-api-openshift-ca\") pod \"93ddea4e-9c77-4dae-8e6b-7e108eebac87\" (UID: \"93ddea4e-9c77-4dae-8e6b-7e108eebac87\") "
Oct 30 19:21:02.749782 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:02.749754    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93ddea4e-9c77-4dae-8e6b-7e108eebac87-maas-api-openshift-ca" pod "93ddea4e-9c77-4dae-8e6b-7e108eebac87" (UID: "93ddea4e-9c77-4dae-8e6b-7e108eebac87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:02.751569 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:02.751543    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93ddea4e-9c77-4dae-8e6b-7e108eebac87-maas-api-xks-ca" pod "93ddea4e-9c77-4dae-8e6b-7e108eebac87" (UID: "93ddea4e-9c77-4dae-8e6b-7e108eebac87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:02.751650 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:02.751550    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ddea4e-9c77-4dae-8e6b-7e108eebac87-kube-api-access-5l5sv" pod "93ddea4e-9c77-4dae-8e6b-7e108eebac87" (UID: "93ddea4e-9c77-4dae-8e6b-7e108eebac87"). InnerVolumeSpecName "kube-api-access-5l5sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:02.850460 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:02.850428    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l5sv\" (UniqueName: \"kubernetes.io/projected/93ddea4e-9c77-4dae-8e6b-7e108eebac87-kube-api-access-5l5sv\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:02.850460 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:02.850459    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93ddea4e-9c77-4dae-8e6b-7e108eebac87-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:02.850650 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:02.850470    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93ddea4e-9c77-4dae-8e6b-7e108eebac87-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:03.517892 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:03.517860    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd"
Oct 30 19:21:03.517892 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:03.517863    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd" event={"ID":"93ddea4e-9c77-4dae-8e6b-7e108eebac87","Type":"ContainerDied","Data":"2d8e3a9b073e64b9d406eb1e6fe8a634be6583c0756eef90fe0c436600909b9f"}
Oct 30 19:21:03.518200 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:03.517918    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d8e3a9b073e64b9d406eb1e6fe8a634be6583c0756eef90fe0c436600909b9f"
Oct 30 19:21:04.299451 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:04.299412    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"]
Oct 30 19:21:04.346160 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:04.346124    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"]
Oct 30 19:21:04.448258 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:04.448223    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:06.185921 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:06.185884    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:10.490758 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:10.490719    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba7f230-llzws"]
Oct 30 19:21:10.491188 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:10.491172    4924 cpu_manager.go:414] "containerMap: removing container" podUID="93ddea4e-9c77-4dae-8e6b-7e108eebac87" containerName="revoke-keys"
Oct 30 19:21:10.491238 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:10.491189    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93ddea4e-9c77-4dae-8e6b-7e108eebac87" containerName="revoke-keys"
Oct 30 19:21:10.491275 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:10.491262    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="93ddea4e-9c77-4dae-8e6b-7e108eebac87" containerName="revoke-keys"
Oct 30 19:21:10.491275 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:10.491274    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb1ef4a-ad74-421f-a5f7-f7e9de5bb421" containerName="istio-proxy"
Oct 30 19:21:10.494108 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:10.494090    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba7f230-llzws"
Oct 30 19:21:10.547815 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:10.547783    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba7f230-llzws"]
Oct 30 19:21:10.591664 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:10.591634    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:21:10.591664 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:10.591656    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:21:10.591880 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:10.591678    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:21:10.591880 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:10.591683    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:21:10.637467 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:10.637434    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a1972f-7378-4c07-9957-ca1ca5e854e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ba7f230-llzws\" (UID: \"29a1972f-7378-4c07-9957-ca1ca5e854e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba7f230-llzws"
Oct 30 19:21:10.637633 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:10.637482    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bd5d\" (UniqueName: \"kubernetes.io/projected/29a1972f-7378-4c07-9957-ca1ca5e854e3-kube-api-access-9bd5d\") pod \"maas-api-revoke-sub-8ba7f230-llzws\" (UID: \"29a1972f-7378-4c07-9957-ca1ca5e854e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba7f230-llzws"
Oct 30 19:21:10.637682 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:10.637639    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a1972f-7378-4c07-9957-ca1ca5e854e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ba7f230-llzws\" (UID: \"29a1972f-7378-4c07-9957-ca1ca5e854e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba7f230-llzws"
Oct 30 19:21:10.738345 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:10.738310    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a1972f-7378-4c07-9957-ca1ca5e854e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ba7f230-llzws\" (UID: \"29a1972f-7378-4c07-9957-ca1ca5e854e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba7f230-llzws"
Oct 30 19:21:10.738513 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:10.738367    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a1972f-7378-4c07-9957-ca1ca5e854e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ba7f230-llzws\" (UID: \"29a1972f-7378-4c07-9957-ca1ca5e854e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba7f230-llzws"
Oct 30 19:21:10.738513 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:10.738410    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bd5d\" (UniqueName: \"kubernetes.io/projected/29a1972f-7378-4c07-9957-ca1ca5e854e3-kube-api-access-9bd5d\") pod \"maas-api-revoke-sub-8ba7f230-llzws\" (UID: \"29a1972f-7378-4c07-9957-ca1ca5e854e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba7f230-llzws"
Oct 30 19:21:10.739157 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:10.739131    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a1972f-7378-4c07-9957-ca1ca5e854e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ba7f230-llzws\" (UID: \"29a1972f-7378-4c07-9957-ca1ca5e854e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba7f230-llzws"
Oct 30 19:21:10.740681 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:10.740659    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a1972f-7378-4c07-9957-ca1ca5e854e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ba7f230-llzws\" (UID: \"29a1972f-7378-4c07-9957-ca1ca5e854e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba7f230-llzws"
Oct 30 19:21:10.793843 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:10.793770    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bd5d\" (UniqueName: \"kubernetes.io/projected/29a1972f-7378-4c07-9957-ca1ca5e854e3-kube-api-access-9bd5d\") pod \"maas-api-revoke-sub-8ba7f230-llzws\" (UID: \"29a1972f-7378-4c07-9957-ca1ca5e854e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba7f230-llzws"
Oct 30 19:21:10.806429 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:10.806404    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba7f230-llzws"
Oct 30 19:21:10.926514 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:10.926477    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94"]
Oct 30 19:21:11.172172 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:11.172145    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba7f230-llzws"]
Oct 30 19:21:11.173512 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:21:11.173489    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a1972f_7378_4c07_9957_ca1ca5e854e3.slice/crio-a7938e39fa0133720e5a12baf575fcab1066cf04ede31895106b8b3da5204331 WatchSource:0}: Error finding container a7938e39fa0133720e5a12baf575fcab1066cf04ede31895106b8b3da5204331: Status 404 returned error can't find the container with id a7938e39fa0133720e5a12baf575fcab1066cf04ede31895106b8b3da5204331
Oct 30 19:21:11.278967 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:11.278928    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:11.566328 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:11.566298    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba7f230-llzws" event={"ID":"29a1972f-7378-4c07-9957-ca1ca5e854e3","Type":"ContainerStarted","Data":"9427cc0d45d7a348c0c9a607032b2fa3aeb3cb9b1e7366f183e2d657f679f79e"}
Oct 30 19:21:11.566724 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:11.566347    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba7f230-llzws" event={"ID":"29a1972f-7378-4c07-9957-ca1ca5e854e3","Type":"ContainerStarted","Data":"a7938e39fa0133720e5a12baf575fcab1066cf04ede31895106b8b3da5204331"}
Oct 30 19:21:11.586842 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:11.586809    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:11.676504 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:11.676451    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba7f230-llzws" podStartSLOduration=1.676438874 podStartE2EDuration="1.676438874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:11.673960896 +0000 UTC m=+2672.951011342" watchObservedRunningTime="2026-10-30 19:21:11.676438874 +0000 UTC m=+2672.953489318"
Oct 30 19:21:12.574735 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:12.574704    4924 generic.go:356] "Generic (PLEG): container finished" podID="29a1972f-7378-4c07-9957-ca1ca5e854e3" containerID="9427cc0d45d7a348c0c9a607032b2fa3aeb3cb9b1e7366f183e2d657f679f79e" exitCode=0
Oct 30 19:21:12.575159 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:12.574787    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba7f230-llzws" event={"ID":"29a1972f-7378-4c07-9957-ca1ca5e854e3","Type":"ContainerDied","Data":"9427cc0d45d7a348c0c9a607032b2fa3aeb3cb9b1e7366f183e2d657f679f79e"}
Oct 30 19:21:13.732376 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:13.732350    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba7f230-llzws"
Oct 30 19:21:13.869640 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:13.869607    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29a1972f-7378-4c07-9957-ca1ca5e854e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a1972f-7378-4c07-9957-ca1ca5e854e3-maas-api-openshift-ca\") pod \"29a1972f-7378-4c07-9957-ca1ca5e854e3\" (UID: \"29a1972f-7378-4c07-9957-ca1ca5e854e3\") "
Oct 30 19:21:13.869862 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:13.869664    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29a1972f-7378-4c07-9957-ca1ca5e854e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a1972f-7378-4c07-9957-ca1ca5e854e3-maas-api-xks-ca\") pod \"29a1972f-7378-4c07-9957-ca1ca5e854e3\" (UID: \"29a1972f-7378-4c07-9957-ca1ca5e854e3\") "
Oct 30 19:21:13.869862 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:13.869742    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29a1972f-7378-4c07-9957-ca1ca5e854e3-kube-api-access-9bd5d\" (UniqueName: \"kubernetes.io/projected/29a1972f-7378-4c07-9957-ca1ca5e854e3-kube-api-access-9bd5d\") pod \"29a1972f-7378-4c07-9957-ca1ca5e854e3\" (UID: \"29a1972f-7378-4c07-9957-ca1ca5e854e3\") "
Oct 30 19:21:13.870058 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:13.870031    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a1972f-7378-4c07-9957-ca1ca5e854e3-maas-api-openshift-ca" pod "29a1972f-7378-4c07-9957-ca1ca5e854e3" (UID: "29a1972f-7378-4c07-9957-ca1ca5e854e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:13.870164 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:13.870147    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a1972f-7378-4c07-9957-ca1ca5e854e3-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:13.872024 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:13.871996    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a1972f-7378-4c07-9957-ca1ca5e854e3-maas-api-xks-ca" pod "29a1972f-7378-4c07-9957-ca1ca5e854e3" (UID: "29a1972f-7378-4c07-9957-ca1ca5e854e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:13.872229 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:13.872209    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a1972f-7378-4c07-9957-ca1ca5e854e3-kube-api-access-9bd5d" pod "29a1972f-7378-4c07-9957-ca1ca5e854e3" (UID: "29a1972f-7378-4c07-9957-ca1ca5e854e3"). InnerVolumeSpecName "kube-api-access-9bd5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:13.971530 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:13.971496    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a1972f-7378-4c07-9957-ca1ca5e854e3-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:13.971530 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:13.971525    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bd5d\" (UniqueName: \"kubernetes.io/projected/29a1972f-7378-4c07-9957-ca1ca5e854e3-kube-api-access-9bd5d\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:14.588944 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:14.588911    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba7f230-llzws"
Oct 30 19:21:14.588944 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:14.588931    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba7f230-llzws" event={"ID":"29a1972f-7378-4c07-9957-ca1ca5e854e3","Type":"ContainerDied","Data":"a7938e39fa0133720e5a12baf575fcab1066cf04ede31895106b8b3da5204331"}
Oct 30 19:21:14.589182 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:14.588980    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7938e39fa0133720e5a12baf575fcab1066cf04ede31895106b8b3da5204331"
Oct 30 19:21:15.712269 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:21:15.712237    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:21:15.927223 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:15.927169    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94" podUID="21e5d49b-a1e4-4232-b823-1d35a77b1ab3" containerName="payload-processing" containerID="cri-o://b681b8d2e7afe4fc4032c08750b3ee91e0058c36c4731839ad5fd22198ee55bf" gracePeriod=25
Oct 30 19:21:19.319033 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.318988    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg" podUID="9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" containerName="main" containerID="cri-o://b93aaa5f51acab31e5206a72c081fcfb623709688ad6cd5a0b8a6a7f1fb25c76" gracePeriod=45
Oct 30 19:21:19.365809 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.365773    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7" podUID="97a3b018-9083-4b7c-af88-6edddc585d67" containerName="main" containerID="cri-o://c8354c5d228abc52be0eeda3d9c4a9a6dcfa6133e3da3e26f846d59ba8c5538c" gracePeriod=45
Oct 30 19:21:19.619992 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.619960    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 19:21:19.622368 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.622343    4924 generic.go:356] "Generic (PLEG): container finished" podID="9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" containerID="b93aaa5f51acab31e5206a72c081fcfb623709688ad6cd5a0b8a6a7f1fb25c76" exitCode=0
Oct 30 19:21:19.622539 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.622385    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"
Oct 30 19:21:19.622539 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.622431    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg" event={"ID":"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e","Type":"ContainerDied","Data":"b93aaa5f51acab31e5206a72c081fcfb623709688ad6cd5a0b8a6a7f1fb25c76"}
Oct 30 19:21:19.622539 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.622476    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg" event={"ID":"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e","Type":"ContainerDied","Data":"812432fe32842d04cbf9b3f6c2024eb64eb9e95c4b6b9fbe988681db6f77f15d"}
Oct 30 19:21:19.622539 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.622502    4924 scope.go:122] "RemoveContainer" containerID="b93aaa5f51acab31e5206a72c081fcfb623709688ad6cd5a0b8a6a7f1fb25c76"
Oct 30 19:21:19.624262 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.624223    4924 generic.go:356] "Generic (PLEG): container finished" podID="97a3b018-9083-4b7c-af88-6edddc585d67" containerID="c8354c5d228abc52be0eeda3d9c4a9a6dcfa6133e3da3e26f846d59ba8c5538c" exitCode=0
Oct 30 19:21:19.624383 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.624281    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7" event={"ID":"97a3b018-9083-4b7c-af88-6edddc585d67","Type":"ContainerDied","Data":"c8354c5d228abc52be0eeda3d9c4a9a6dcfa6133e3da3e26f846d59ba8c5538c"}
Oct 30 19:21:19.632348 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.632327    4924 scope.go:122] "RemoveContainer" containerID="b93aaa5f51acab31e5206a72c081fcfb623709688ad6cd5a0b8a6a7f1fb25c76"
Oct 30 19:21:19.632692 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:21:19.632665    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b93aaa5f51acab31e5206a72c081fcfb623709688ad6cd5a0b8a6a7f1fb25c76\": container with ID starting with b93aaa5f51acab31e5206a72c081fcfb623709688ad6cd5a0b8a6a7f1fb25c76 not found: ID does not exist" containerID="b93aaa5f51acab31e5206a72c081fcfb623709688ad6cd5a0b8a6a7f1fb25c76"
Oct 30 19:21:19.632789 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.632702    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b93aaa5f51acab31e5206a72c081fcfb623709688ad6cd5a0b8a6a7f1fb25c76"} err="failed to get container status \"b93aaa5f51acab31e5206a72c081fcfb623709688ad6cd5a0b8a6a7f1fb25c76\": rpc error: code = NotFound desc = could not find container \"b93aaa5f51acab31e5206a72c081fcfb623709688ad6cd5a0b8a6a7f1fb25c76\": container with ID starting with b93aaa5f51acab31e5206a72c081fcfb623709688ad6cd5a0b8a6a7f1fb25c76 not found: ID does not exist"
Oct 30 19:21:19.650827 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.650803    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 19:21:19.731605 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.731536    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-home\" (UniqueName: \"kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-home\") pod \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\" (UID: \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\") "
Oct 30 19:21:19.731605 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.731587    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-kube-api-access-pjx2h\" (UniqueName: \"kubernetes.io/projected/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-kube-api-access-pjx2h\") pod \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\" (UID: \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\") "
Oct 30 19:21:19.731938 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.731675    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-model-cache\") pod \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\" (UID: \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\") "
Oct 30 19:21:19.731938 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.731766    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-tls-certs\" (UniqueName: \"kubernetes.io/secret/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-tls-certs\") pod \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\" (UID: \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\") "
Oct 30 19:21:19.731938 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.731784    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-home" pod "9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" (UID: "9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:19.731938 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.731851    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-tmp-dir\") pod \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\" (UID: \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\") "
Oct 30 19:21:19.731938 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.731892    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-dshm\") pod \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\" (UID: \"9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e\") "
Oct 30 19:21:19.732268 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.732247    4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-home\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:19.732536 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.732496    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-model-cache" pod "9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" (UID: "9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:19.732689 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.732574    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-tmp-dir" pod "9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" (UID: "9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:19.734462 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.734437    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-tls-certs" pod "9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" (UID: "9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:19.734736 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.734715    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-dshm" pod "9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" (UID: "9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:19.734825 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.734715    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-kube-api-access-pjx2h" pod "9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" (UID: "9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e"). InnerVolumeSpecName "kube-api-access-pjx2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:19.833125 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.833024    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97a3b018-9083-4b7c-af88-6edddc585d67-tls-certs\" (UniqueName: \"kubernetes.io/secret/97a3b018-9083-4b7c-af88-6edddc585d67-tls-certs\") pod \"97a3b018-9083-4b7c-af88-6edddc585d67\" (UID: \"97a3b018-9083-4b7c-af88-6edddc585d67\") "
Oct 30 19:21:19.833125 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.833121    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-model-cache\") pod \"97a3b018-9083-4b7c-af88-6edddc585d67\" (UID: \"97a3b018-9083-4b7c-af88-6edddc585d67\") "
Oct 30 19:21:19.833317 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.833188    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97a3b018-9083-4b7c-af88-6edddc585d67-kube-api-access-cnbk6\" (UniqueName: \"kubernetes.io/projected/97a3b018-9083-4b7c-af88-6edddc585d67-kube-api-access-cnbk6\") pod \"97a3b018-9083-4b7c-af88-6edddc585d67\" (UID: \"97a3b018-9083-4b7c-af88-6edddc585d67\") "
Oct 30 19:21:19.833317 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.833216    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-tmp-dir\") pod \"97a3b018-9083-4b7c-af88-6edddc585d67\" (UID: \"97a3b018-9083-4b7c-af88-6edddc585d67\") "
Oct 30 19:21:19.833317 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.833265    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-home\" (UniqueName: \"kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-home\") pod \"97a3b018-9083-4b7c-af88-6edddc585d67\" (UID: \"97a3b018-9083-4b7c-af88-6edddc585d67\") "
Oct 30 19:21:19.833317 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.833298    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-dshm\") pod \"97a3b018-9083-4b7c-af88-6edddc585d67\" (UID: \"97a3b018-9083-4b7c-af88-6edddc585d67\") "
Oct 30 19:21:19.833523 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.833425    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-model-cache" pod "97a3b018-9083-4b7c-af88-6edddc585d67" (UID: "97a3b018-9083-4b7c-af88-6edddc585d67"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:19.833610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.833586    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-tmp-dir" pod "97a3b018-9083-4b7c-af88-6edddc585d67" (UID: "97a3b018-9083-4b7c-af88-6edddc585d67"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:19.833668 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.833604    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-home" pod "97a3b018-9083-4b7c-af88-6edddc585d67" (UID: "97a3b018-9083-4b7c-af88-6edddc585d67"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:19.833708 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.833676    4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-tls-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:19.833708 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.833700    4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-tmp-dir\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:19.833825 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.833716    4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-dshm\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:19.833825 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.833731    4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-model-cache\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:19.833825 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.833749    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjx2h\" (UniqueName: \"kubernetes.io/projected/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-kube-api-access-pjx2h\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:19.833825 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.833764    4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e-model-cache\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:19.835257 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.835239    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a3b018-9083-4b7c-af88-6edddc585d67-kube-api-access-cnbk6" pod "97a3b018-9083-4b7c-af88-6edddc585d67" (UID: "97a3b018-9083-4b7c-af88-6edddc585d67"). InnerVolumeSpecName "kube-api-access-cnbk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:19.835374 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.835261    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97a3b018-9083-4b7c-af88-6edddc585d67-tls-certs" pod "97a3b018-9083-4b7c-af88-6edddc585d67" (UID: "97a3b018-9083-4b7c-af88-6edddc585d67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:21:19.835374 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.835274    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-dshm" pod "97a3b018-9083-4b7c-af88-6edddc585d67" (UID: "97a3b018-9083-4b7c-af88-6edddc585d67"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:19.934513 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.934480    4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97a3b018-9083-4b7c-af88-6edddc585d67-tls-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:19.934716 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.934515    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnbk6\" (UniqueName: \"kubernetes.io/projected/97a3b018-9083-4b7c-af88-6edddc585d67-kube-api-access-cnbk6\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:19.934716 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.934534    4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-tmp-dir\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:19.934716 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.934548    4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-home\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:19.934716 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:19.934562    4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97a3b018-9083-4b7c-af88-6edddc585d67-dshm\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:20.038052 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:20.038018    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"]
Oct 30 19:21:20.042956 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:20.042922    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg"]
Oct 30 19:21:20.611174 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:20.611122    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f5705f/distinct-w0-f5705f-kserve-5585ddc876-4blrg" podUID="9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Oct 30 19:21:20.634244 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:20.634220    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"
Oct 30 19:21:20.634403 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:20.634254    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7" event={"ID":"97a3b018-9083-4b7c-af88-6edddc585d67","Type":"ContainerDied","Data":"a2c4f3ed50c3532bcbbd76c5d47d71543b05463eed799826c42ca38bca46b7e4"}
Oct 30 19:21:20.634403 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:20.634307    4924 scope.go:122] "RemoveContainer" containerID="c8354c5d228abc52be0eeda3d9c4a9a6dcfa6133e3da3e26f846d59ba8c5538c"
Oct 30 19:21:20.791363 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:20.791326    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"]
Oct 30 19:21:20.795761 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:20.795734    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f5705f/premium-w0-f5705f-kserve-7854c57b49-xrck7"]
Oct 30 19:21:20.833383 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:20.833358    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" path="/var/lib/kubelet/pods/9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e/volumes"
Oct 30 19:21:20.833824 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:20.833810    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97a3b018-9083-4b7c-af88-6edddc585d67" path="/var/lib/kubelet/pods/97a3b018-9083-4b7c-af88-6edddc585d67/volumes"
Oct 30 19:21:22.428938 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:22.428899    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94" podUID="21e5d49b-a1e4-4232-b823-1d35a77b1ab3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:21:25.427367 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:25.427333    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:26.955523 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:26.955482    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:27.247512 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:27.247371    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:27.464298 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:27.464262    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd"]
Oct 30 19:21:27.510746 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:27.510673    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f5705f-2c2jd"]
Oct 30 19:21:27.893399 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:27.893365    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:28.833673 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:28.833631    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93ddea4e-9c77-4dae-8e6b-7e108eebac87" path="/var/lib/kubelet/pods/93ddea4e-9c77-4dae-8e6b-7e108eebac87/volumes"
Oct 30 19:21:31.059123 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:31.059088    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb621801-s69jf"]
Oct 30 19:21:31.106347 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:31.106313    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb621801-s69jf"]
Oct 30 19:21:31.108300 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:31.108290    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a469b055-zdg5v"]
Oct 30 19:21:31.110039 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:31.110017    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a469b055-zdg5v"]
Oct 30 19:21:32.428839 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:32.428803    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94" podUID="21e5d49b-a1e4-4232-b823-1d35a77b1ab3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:21:32.833388 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:32.833313    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37f730bf-6bd9-45f3-b45c-e35ebff3cb39" path="/var/lib/kubelet/pods/37f730bf-6bd9-45f3-b45c-e35ebff3cb39/volumes"
Oct 30 19:21:32.833848 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:32.833832    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="854e149f-ec52-42cd-8a49-ac4943ee6163" path="/var/lib/kubelet/pods/854e149f-ec52-42cd-8a49-ac4943ee6163/volumes"
Oct 30 19:21:35.282017 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:35.281981    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:35.654267 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:35.654232    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"]
Oct 30 19:21:35.654465 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:35.654434    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv" podUID="ccddacc3-f62d-477a-9c8b-9ead25b237c4" containerName="istio-proxy" containerID="cri-o://000e132108360b6e8a9560dc833ccabdceb7c3409d69cc92d224cb02fd118fb2" gracePeriod=30
Oct 30 19:21:35.656424 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:35.656388    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:35.923935 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:35.923856    4924 patch_prober.go:29] interesting pod/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:21:35.923935 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:35.923925    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv" podUID="ccddacc3-f62d-477a-9c8b-9ead25b237c4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:21:35.969896 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:35.969853    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:36.183185 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:36.183116    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94"
Oct 30 19:21:36.286020 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:36.285984    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21e5d49b-a1e4-4232-b823-1d35a77b1ab3-kube-api-access-9jbn9\" (UniqueName: \"kubernetes.io/projected/21e5d49b-a1e4-4232-b823-1d35a77b1ab3-kube-api-access-9jbn9\") pod \"21e5d49b-a1e4-4232-b823-1d35a77b1ab3\" (UID: \"21e5d49b-a1e4-4232-b823-1d35a77b1ab3\") "
Oct 30 19:21:36.286478 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:36.286058    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21e5d49b-a1e4-4232-b823-1d35a77b1ab3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e5d49b-a1e4-4232-b823-1d35a77b1ab3-plugins-config-volume\") pod \"21e5d49b-a1e4-4232-b823-1d35a77b1ab3\" (UID: \"21e5d49b-a1e4-4232-b823-1d35a77b1ab3\") "
Oct 30 19:21:36.286478 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:36.286400    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21e5d49b-a1e4-4232-b823-1d35a77b1ab3-plugins-config-volume" pod "21e5d49b-a1e4-4232-b823-1d35a77b1ab3" (UID: "21e5d49b-a1e4-4232-b823-1d35a77b1ab3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:36.288141 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:36.288113    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e5d49b-a1e4-4232-b823-1d35a77b1ab3-kube-api-access-9jbn9" pod "21e5d49b-a1e4-4232-b823-1d35a77b1ab3" (UID: "21e5d49b-a1e4-4232-b823-1d35a77b1ab3"). InnerVolumeSpecName "kube-api-access-9jbn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:36.387223 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:36.387183    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jbn9\" (UniqueName: \"kubernetes.io/projected/21e5d49b-a1e4-4232-b823-1d35a77b1ab3-kube-api-access-9jbn9\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:36.387223 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:36.387216    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e5d49b-a1e4-4232-b823-1d35a77b1ab3-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:36.729214 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:36.729179    4924 generic.go:356] "Generic (PLEG): container finished" podID="21e5d49b-a1e4-4232-b823-1d35a77b1ab3" containerID="b681b8d2e7afe4fc4032c08750b3ee91e0058c36c4731839ad5fd22198ee55bf" exitCode=0
Oct 30 19:21:36.729416 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:36.729223    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94"
Oct 30 19:21:36.729416 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:36.729267    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94" event={"ID":"21e5d49b-a1e4-4232-b823-1d35a77b1ab3","Type":"ContainerDied","Data":"b681b8d2e7afe4fc4032c08750b3ee91e0058c36c4731839ad5fd22198ee55bf"}
Oct 30 19:21:36.729416 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:36.729318    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94" event={"ID":"21e5d49b-a1e4-4232-b823-1d35a77b1ab3","Type":"ContainerDied","Data":"2b95dae47c647476ed0acb7a99467bd2a626e66b87e5ee44c663cf323aaa987f"}
Oct 30 19:21:36.729416 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:36.729346    4924 scope.go:122] "RemoveContainer" containerID="b681b8d2e7afe4fc4032c08750b3ee91e0058c36c4731839ad5fd22198ee55bf"
Oct 30 19:21:36.739538 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:36.739520    4924 scope.go:122] "RemoveContainer" containerID="b681b8d2e7afe4fc4032c08750b3ee91e0058c36c4731839ad5fd22198ee55bf"
Oct 30 19:21:36.739814 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:21:36.739797    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b681b8d2e7afe4fc4032c08750b3ee91e0058c36c4731839ad5fd22198ee55bf\": container with ID starting with b681b8d2e7afe4fc4032c08750b3ee91e0058c36c4731839ad5fd22198ee55bf not found: ID does not exist" containerID="b681b8d2e7afe4fc4032c08750b3ee91e0058c36c4731839ad5fd22198ee55bf"
Oct 30 19:21:36.739857 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:36.739821    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b681b8d2e7afe4fc4032c08750b3ee91e0058c36c4731839ad5fd22198ee55bf"} err="failed to get container status \"b681b8d2e7afe4fc4032c08750b3ee91e0058c36c4731839ad5fd22198ee55bf\": rpc error: code = NotFound desc = could not find container \"b681b8d2e7afe4fc4032c08750b3ee91e0058c36c4731839ad5fd22198ee55bf\": container with ID starting with b681b8d2e7afe4fc4032c08750b3ee91e0058c36c4731839ad5fd22198ee55bf not found: ID does not exist"
Oct 30 19:21:36.885392 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:36.885358    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94"]
Oct 30 19:21:36.890314 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:36.890286    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f5705f-5d6f5b6cbd-fmp94"]
Oct 30 19:21:38.833283 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:38.833248    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21e5d49b-a1e4-4232-b823-1d35a77b1ab3" path="/var/lib/kubelet/pods/21e5d49b-a1e4-4232-b823-1d35a77b1ab3/volumes"
Oct 30 19:21:39.316672 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:39.316640    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 19:21:39.317799 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:39.317774    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 19:21:40.909033 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:40.909011    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 19:21:41.036376 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.036295    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-data\") pod \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") "
Oct 30 19:21:41.036376 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.036358    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-envoy\") pod \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") "
Oct 30 19:21:41.036589 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.036452    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccddacc3-f62d-477a-9c8b-9ead25b237c4-kube-api-access-bh62c\" (UniqueName: \"kubernetes.io/projected/ccddacc3-f62d-477a-9c8b-9ead25b237c4-kube-api-access-bh62c\") pod \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") "
Oct 30 19:21:41.036589 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.036483    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-credential-socket\") pod \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") "
Oct 30 19:21:41.036589 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.036516    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istiod-ca-cert\") pod \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") "
Oct 30 19:21:41.036589 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.036548    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-token\" (UniqueName: \"kubernetes.io/projected/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-token\") pod \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") "
Oct 30 19:21:41.036589 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.036585    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-workload-certs\") pod \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") "
Oct 30 19:21:41.036864 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.036610    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-data" pod "ccddacc3-f62d-477a-9c8b-9ead25b237c4" (UID: "ccddacc3-f62d-477a-9c8b-9ead25b237c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:41.036864 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.036619    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-workload-socket\") pod \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") "
Oct 30 19:21:41.036864 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.036703    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-podinfo\") pod \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\" (UID: \"ccddacc3-f62d-477a-9c8b-9ead25b237c4\") "
Oct 30 19:21:41.036864 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.036819    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-credential-socket" pod "ccddacc3-f62d-477a-9c8b-9ead25b237c4" (UID: "ccddacc3-f62d-477a-9c8b-9ead25b237c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:41.037120 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.036869    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-workload-socket" pod "ccddacc3-f62d-477a-9c8b-9ead25b237c4" (UID: "ccddacc3-f62d-477a-9c8b-9ead25b237c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:41.037120 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.036914    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istiod-ca-cert" pod "ccddacc3-f62d-477a-9c8b-9ead25b237c4" (UID: "ccddacc3-f62d-477a-9c8b-9ead25b237c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:21:41.037120 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.036992    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-workload-certs" pod "ccddacc3-f62d-477a-9c8b-9ead25b237c4" (UID: "ccddacc3-f62d-477a-9c8b-9ead25b237c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:41.037304 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.037148    4924 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-credential-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:41.037304 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.037169    4924 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istiod-ca-cert\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:41.037304 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.037186    4924 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-workload-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:41.037304 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.037199    4924 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-workload-socket\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:41.037304 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.037212    4924 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-data\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:41.038839 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.038813    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccddacc3-f62d-477a-9c8b-9ead25b237c4-kube-api-access-bh62c" pod "ccddacc3-f62d-477a-9c8b-9ead25b237c4" (UID: "ccddacc3-f62d-477a-9c8b-9ead25b237c4"). InnerVolumeSpecName "kube-api-access-bh62c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:41.038956 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.038842    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-envoy" pod "ccddacc3-f62d-477a-9c8b-9ead25b237c4" (UID: "ccddacc3-f62d-477a-9c8b-9ead25b237c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:21:41.038956 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.038850    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-token" pod "ccddacc3-f62d-477a-9c8b-9ead25b237c4" (UID: "ccddacc3-f62d-477a-9c8b-9ead25b237c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:21:41.038956 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.038926    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-podinfo" pod "ccddacc3-f62d-477a-9c8b-9ead25b237c4" (UID: "ccddacc3-f62d-477a-9c8b-9ead25b237c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:21:41.137819 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.137787    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh62c\" (UniqueName: \"kubernetes.io/projected/ccddacc3-f62d-477a-9c8b-9ead25b237c4-kube-api-access-bh62c\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:41.137819 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.137818    4924 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-token\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:41.138057 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.137831    4924 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-podinfo\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:41.138057 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.137843    4924 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccddacc3-f62d-477a-9c8b-9ead25b237c4-istio-envoy\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:21:41.411723 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.411690    4924 scope.go:122] "RemoveContainer" containerID="8cec6bf9812cf618d47bd9ceba66988ff49d92d1c170ebcb0ffd9e71abd9081c"
Oct 30 19:21:41.422456 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.422432    4924 scope.go:122] "RemoveContainer" containerID="09cbd77013c968922d299888c5accb478bd453198ca4d16fda4be5dd5582bf19"
Oct 30 19:21:41.430845 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.430821    4924 scope.go:122] "RemoveContainer" containerID="9decf978143b0f918adb02392bedf16ab768273b91f846999894c496760ddf60"
Oct 30 19:21:41.439263 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.439240    4924 scope.go:122] "RemoveContainer" containerID="83390c9f2aad38007fe670e0fa3ee6b64baee1b78c951cb73e48bf62e0164ae7"
Oct 30 19:21:41.448256 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.448227    4924 scope.go:122] "RemoveContainer" containerID="24edc2c8cc0a4bd1289895d4ca56ccc2dfe7b7f1310385b5f6bea38b7d9b1871"
Oct 30 19:21:41.456854 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.456835    4924 scope.go:122] "RemoveContainer" containerID="000e132108360b6e8a9560dc833ccabdceb7c3409d69cc92d224cb02fd118fb2"
Oct 30 19:21:41.465262 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.465243    4924 scope.go:122] "RemoveContainer" containerID="0b18634c91f2b5568c37c7f37fc68e58f4b0b2cc4e49be27435a37ca33434e14"
Oct 30 19:21:41.473194 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.473173    4924 scope.go:122] "RemoveContainer" containerID="289433759f06be19a725938fb04d2bc9b1091b5e234674be23844d24a5cdc034"
Oct 30 19:21:41.480898 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.480879    4924 scope.go:122] "RemoveContainer" containerID="3d3ed449af66eecea3164a9e6c2afa82b207f1ca5b430207360288935c5744ca"
Oct 30 19:21:41.488699 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.488671    4924 scope.go:122] "RemoveContainer" containerID="ff16e300450d66861b648b25d698f026cc50115abfb9610a2be55ac0e71efae2"
Oct 30 19:21:41.763453 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.763368    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv" event={"ID":"ccddacc3-f62d-477a-9c8b-9ead25b237c4","Type":"ContainerDied","Data":"000e132108360b6e8a9560dc833ccabdceb7c3409d69cc92d224cb02fd118fb2"}
Oct 30 19:21:41.763453 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.763430    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv" event={"ID":"ccddacc3-f62d-477a-9c8b-9ead25b237c4","Type":"ContainerDied","Data":"b1aac27ae395ea94c97ce83835f80d4ea16eda598ac9e80adb4a71dc488a39ec"}
Oct 30 19:21:41.763656 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:41.763508    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"
Oct 30 19:21:42.043735 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:42.043645    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"]
Oct 30 19:21:42.049469 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:42.049441    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f5705f-openshift-default-64b647b647-tpfvv"]
Oct 30 19:21:42.626445 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:42.626390    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:42.836910 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:42.836869    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccddacc3-f62d-477a-9c8b-9ead25b237c4" path="/var/lib/kubelet/pods/ccddacc3-f62d-477a-9c8b-9ead25b237c4/volumes"
Oct 30 19:21:42.981521 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:42.981415    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:43.422850 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:43.422815    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:43.744614 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:43.744475    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:45.711429 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:21:45.711396    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:21:48.834164 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:48.834130    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:56.056035 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:56.055998    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:56.377930 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:56.377898    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:58.048386 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:58.048352    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baeda332-g5dvx"]
Oct 30 19:21:58.094915 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:58.094879    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7da0a81c-92hnt"]
Oct 30 19:21:58.096858 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:58.096834    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baeda332-g5dvx"]
Oct 30 19:21:58.098429 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:58.098412    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7da0a81c-92hnt"]
Oct 30 19:21:58.833159 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:58.833127    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37d1db7a-108c-4972-8223-3ed0a626a6dc" path="/var/lib/kubelet/pods/37d1db7a-108c-4972-8223-3ed0a626a6dc/volumes"
Oct 30 19:21:58.833575 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:21:58.833562    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb3461be-6387-4cf8-8d71-082737fdd0d8" path="/var/lib/kubelet/pods/fb3461be-6387-4cf8-8d71-082737fdd0d8/volumes"
Oct 30 19:22:00.886807 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:00.886776    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:01.243897 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:01.243790    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:04.047237 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:04.047207    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37d4b500-hpn5f"]
Oct 30 19:22:04.093932 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:04.093893    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15294a79-vdrpp"]
Oct 30 19:22:04.095634 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:04.095617    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37d4b500-hpn5f"]
Oct 30 19:22:04.097089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:04.097057    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15294a79-vdrpp"]
Oct 30 19:22:04.834422 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:04.834389    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04617682-2d5b-421c-82c4-e7e60c01edbf" path="/var/lib/kubelet/pods/04617682-2d5b-421c-82c4-e7e60c01edbf/volumes"
Oct 30 19:22:04.834808 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:04.834793    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96230ce5-7a7e-48f9-a87f-1b786d7f8730" path="/var/lib/kubelet/pods/96230ce5-7a7e-48f9-a87f-1b786d7f8730/volumes"
Oct 30 19:22:07.514226 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.514185    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-825zc"]
Oct 30 19:22:07.514707 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.514596    4924 cpu_manager.go:414] "containerMap: removing container" podUID="21e5d49b-a1e4-4232-b823-1d35a77b1ab3" containerName="payload-processing"
Oct 30 19:22:07.514707 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.514606    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e5d49b-a1e4-4232-b823-1d35a77b1ab3" containerName="payload-processing"
Oct 30 19:22:07.514707 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.514619    4924 cpu_manager.go:414] "containerMap: removing container" podUID="ccddacc3-f62d-477a-9c8b-9ead25b237c4" containerName="istio-proxy"
Oct 30 19:22:07.514707 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.514624    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccddacc3-f62d-477a-9c8b-9ead25b237c4" containerName="istio-proxy"
Oct 30 19:22:07.514707 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.514635    4924 cpu_manager.go:414] "containerMap: removing container" podUID="9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" containerName="main"
Oct 30 19:22:07.514707 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.514640    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" containerName="main"
Oct 30 19:22:07.514707 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.514647    4924 cpu_manager.go:414] "containerMap: removing container" podUID="29a1972f-7378-4c07-9957-ca1ca5e854e3" containerName="revoke-keys"
Oct 30 19:22:07.514707 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.514652    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29a1972f-7378-4c07-9957-ca1ca5e854e3" containerName="revoke-keys"
Oct 30 19:22:07.514707 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.514669    4924 cpu_manager.go:414] "containerMap: removing container" podUID="97a3b018-9083-4b7c-af88-6edddc585d67" containerName="main"
Oct 30 19:22:07.514707 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.514673    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a3b018-9083-4b7c-af88-6edddc585d67" containerName="main"
Oct 30 19:22:07.515035 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.514745    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e5d49b-a1e4-4232-b823-1d35a77b1ab3" containerName="payload-processing"
Oct 30 19:22:07.515035 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.514753    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccddacc3-f62d-477a-9c8b-9ead25b237c4" containerName="istio-proxy"
Oct 30 19:22:07.515035 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.514759    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="9176bed1-9bd8-4a37-ba3b-bc20bcfc8b5e" containerName="main"
Oct 30 19:22:07.515035 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.514765    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="29a1972f-7378-4c07-9957-ca1ca5e854e3" containerName="revoke-keys"
Oct 30 19:22:07.515035 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.514773    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="97a3b018-9083-4b7c-af88-6edddc585d67" containerName="main"
Oct 30 19:22:07.515035 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.514779    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccddacc3-f62d-477a-9c8b-9ead25b237c4" containerName="istio-proxy"
Oct 30 19:22:07.517701 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.517681    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-825zc"
Oct 30 19:22:07.532721 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.532693    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq48p\" (UniqueName: \"kubernetes.io/projected/9a6a2041-f7ce-4f30-a120-cd3b77da55a8-kube-api-access-gq48p\") pod \"maas-api-revoke-sub-85237e45-825zc\" (UID: \"9a6a2041-f7ce-4f30-a120-cd3b77da55a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-825zc"
Oct 30 19:22:07.532856 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.532802    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6a2041-f7ce-4f30-a120-cd3b77da55a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-825zc\" (UID: \"9a6a2041-f7ce-4f30-a120-cd3b77da55a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-825zc"
Oct 30 19:22:07.532916 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.532878    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6a2041-f7ce-4f30-a120-cd3b77da55a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-825zc\" (UID: \"9a6a2041-f7ce-4f30-a120-cd3b77da55a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-825zc"
Oct 30 19:22:07.567627 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.567591    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:22:07.567627 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.567603    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:22:07.613939 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.613908    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:22:07.613939 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.613919    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:22:07.616251 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.616220    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-825zc"]
Oct 30 19:22:07.633905 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.633875    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6a2041-f7ce-4f30-a120-cd3b77da55a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-825zc\" (UID: \"9a6a2041-f7ce-4f30-a120-cd3b77da55a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-825zc"
Oct 30 19:22:07.634031 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.633928    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq48p\" (UniqueName: \"kubernetes.io/projected/9a6a2041-f7ce-4f30-a120-cd3b77da55a8-kube-api-access-gq48p\") pod \"maas-api-revoke-sub-85237e45-825zc\" (UID: \"9a6a2041-f7ce-4f30-a120-cd3b77da55a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-825zc"
Oct 30 19:22:07.634031 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.633983    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6a2041-f7ce-4f30-a120-cd3b77da55a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-825zc\" (UID: \"9a6a2041-f7ce-4f30-a120-cd3b77da55a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-825zc"
Oct 30 19:22:07.634562 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.634537    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6a2041-f7ce-4f30-a120-cd3b77da55a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-825zc\" (UID: \"9a6a2041-f7ce-4f30-a120-cd3b77da55a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-825zc"
Oct 30 19:22:07.636359 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.636338    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6a2041-f7ce-4f30-a120-cd3b77da55a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-825zc\" (UID: \"9a6a2041-f7ce-4f30-a120-cd3b77da55a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-825zc"
Oct 30 19:22:07.688799 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.688767    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq48p\" (UniqueName: \"kubernetes.io/projected/9a6a2041-f7ce-4f30-a120-cd3b77da55a8-kube-api-access-gq48p\") pod \"maas-api-revoke-sub-85237e45-825zc\" (UID: \"9a6a2041-f7ce-4f30-a120-cd3b77da55a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-825zc"
Oct 30 19:22:07.830868 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:07.830789    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-825zc"
Oct 30 19:22:08.193143 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:08.193120    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-825zc"]
Oct 30 19:22:08.195423 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:22:08.195392    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6a2041_f7ce_4f30_a120_cd3b77da55a8.slice/crio-fd15e245e00ac38aa120b09061957def772d5e18b02d6f2c624637056c5d8371 WatchSource:0}: Error finding container fd15e245e00ac38aa120b09061957def772d5e18b02d6f2c624637056c5d8371: Status 404 returned error can't find the container with id fd15e245e00ac38aa120b09061957def772d5e18b02d6f2c624637056c5d8371
Oct 30 19:22:08.930642 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:08.930607    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-825zc" event={"ID":"9a6a2041-f7ce-4f30-a120-cd3b77da55a8","Type":"ContainerStarted","Data":"cdf2e281254a892508470c5332bafcfc0a8e3f4151147e9a90731fb7e929ade9"}
Oct 30 19:22:08.931044 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:08.930663    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-825zc" event={"ID":"9a6a2041-f7ce-4f30-a120-cd3b77da55a8","Type":"ContainerStarted","Data":"fd15e245e00ac38aa120b09061957def772d5e18b02d6f2c624637056c5d8371"}
Oct 30 19:22:09.041335 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:09.041286    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-825zc" podStartSLOduration=2.041272039 podStartE2EDuration="2.041272039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:09.038539662 +0000 UTC m=+2730.315590107" watchObservedRunningTime="2026-10-30 19:22:09.041272039 +0000 UTC m=+2730.318322481"
Oct 30 19:22:09.937046 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:09.937014    4924 generic.go:356] "Generic (PLEG): container finished" podID="9a6a2041-f7ce-4f30-a120-cd3b77da55a8" containerID="cdf2e281254a892508470c5332bafcfc0a8e3f4151147e9a90731fb7e929ade9" exitCode=0
Oct 30 19:22:09.937467 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:09.937113    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-825zc" event={"ID":"9a6a2041-f7ce-4f30-a120-cd3b77da55a8","Type":"ContainerDied","Data":"cdf2e281254a892508470c5332bafcfc0a8e3f4151147e9a90731fb7e929ade9"}
Oct 30 19:22:11.068728 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:11.068703    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-825zc"
Oct 30 19:22:11.170132 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:11.170100    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a6a2041-f7ce-4f30-a120-cd3b77da55a8-kube-api-access-gq48p\" (UniqueName: \"kubernetes.io/projected/9a6a2041-f7ce-4f30-a120-cd3b77da55a8-kube-api-access-gq48p\") pod \"9a6a2041-f7ce-4f30-a120-cd3b77da55a8\" (UID: \"9a6a2041-f7ce-4f30-a120-cd3b77da55a8\") "
Oct 30 19:22:11.170321 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:11.170168    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a6a2041-f7ce-4f30-a120-cd3b77da55a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6a2041-f7ce-4f30-a120-cd3b77da55a8-maas-api-xks-ca\") pod \"9a6a2041-f7ce-4f30-a120-cd3b77da55a8\" (UID: \"9a6a2041-f7ce-4f30-a120-cd3b77da55a8\") "
Oct 30 19:22:11.170321 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:11.170217    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a6a2041-f7ce-4f30-a120-cd3b77da55a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6a2041-f7ce-4f30-a120-cd3b77da55a8-maas-api-openshift-ca\") pod \"9a6a2041-f7ce-4f30-a120-cd3b77da55a8\" (UID: \"9a6a2041-f7ce-4f30-a120-cd3b77da55a8\") "
Oct 30 19:22:11.170621 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:11.170591    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a6a2041-f7ce-4f30-a120-cd3b77da55a8-maas-api-openshift-ca" pod "9a6a2041-f7ce-4f30-a120-cd3b77da55a8" (UID: "9a6a2041-f7ce-4f30-a120-cd3b77da55a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:11.172276 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:11.172244    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a6a2041-f7ce-4f30-a120-cd3b77da55a8-maas-api-xks-ca" pod "9a6a2041-f7ce-4f30-a120-cd3b77da55a8" (UID: "9a6a2041-f7ce-4f30-a120-cd3b77da55a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:22:11.172388 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:11.172278    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a6a2041-f7ce-4f30-a120-cd3b77da55a8-kube-api-access-gq48p" pod "9a6a2041-f7ce-4f30-a120-cd3b77da55a8" (UID: "9a6a2041-f7ce-4f30-a120-cd3b77da55a8"). InnerVolumeSpecName "kube-api-access-gq48p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:11.271543 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:11.271463    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq48p\" (UniqueName: \"kubernetes.io/projected/9a6a2041-f7ce-4f30-a120-cd3b77da55a8-kube-api-access-gq48p\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:11.271543 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:11.271492    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6a2041-f7ce-4f30-a120-cd3b77da55a8-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:11.271543 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:11.271502    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6a2041-f7ce-4f30-a120-cd3b77da55a8-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:22:11.951839 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:11.951805    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-825zc"
Oct 30 19:22:11.951839 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:11.951816    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-825zc" event={"ID":"9a6a2041-f7ce-4f30-a120-cd3b77da55a8","Type":"ContainerDied","Data":"fd15e245e00ac38aa120b09061957def772d5e18b02d6f2c624637056c5d8371"}
Oct 30 19:22:11.952176 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:11.951875    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd15e245e00ac38aa120b09061957def772d5e18b02d6f2c624637056c5d8371"
Oct 30 19:22:15.050833 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:15.050794    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a41e9e11-9wrkt"]
Oct 30 19:22:15.097506 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:15.097473    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a41e9e11-9wrkt"]
Oct 30 19:22:15.710236 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:22:15.710204    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:22:16.833648 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:16.833615    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="996da56a-ede7-4041-b0c4-1d5f10f1b659" path="/var/lib/kubelet/pods/996da56a-ede7-4041-b0c4-1d5f10f1b659/volumes"
Oct 30 19:22:19.539001 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:19.538967    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:22.326247 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:22.326208    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:38.138673 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:38.138639    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8260ef6-n25cb"]
Oct 30 19:22:38.185388 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:38.185347    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddb6c82f-g4889"]
Oct 30 19:22:38.186933 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:38.186915    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddb6c82f-g4889"]
Oct 30 19:22:38.188348 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:38.188327    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8260ef6-n25cb"]
Oct 30 19:22:38.832701 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:38.832670    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03af0d78-56e2-44ab-9e4f-32f8aa9363a8" path="/var/lib/kubelet/pods/03af0d78-56e2-44ab-9e4f-32f8aa9363a8/volumes"
Oct 30 19:22:38.833103 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:38.833088    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f98c2b6d-a534-43ef-bee4-780bf3f30539" path="/var/lib/kubelet/pods/f98c2b6d-a534-43ef-bee4-780bf3f30539/volumes"
Oct 30 19:22:41.556242 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:41.556206    4924 scope.go:122] "RemoveContainer" containerID="1faaa1146693c3f88a3cb56132992e250d08510ae7b30c5b28459ba4fa3225bf"
Oct 30 19:22:41.565400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:41.565380    4924 scope.go:122] "RemoveContainer" containerID="8a880f53d849a92d172f35f8a62a545d97ea337aaf68457fccfb6f4bd19c3210"
Oct 30 19:22:41.573608 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:41.573582    4924 scope.go:122] "RemoveContainer" containerID="6dbdc5fa3ca4a21a608e4f6ef5f869506abc70545ceba4b71c32897d1e86a661"
Oct 30 19:22:41.581590 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:41.581571    4924 scope.go:122] "RemoveContainer" containerID="dad5539a375292c5562b07a3011e9c51f81da5e4cc617af1a380f38db1d8424e"
Oct 30 19:22:41.589147 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:41.589129    4924 scope.go:122] "RemoveContainer" containerID="88a3779d8e6f0f7d5b5d953e8c6570dcb992e25b37bd8ad367bffd023eef3e62"
Oct 30 19:22:43.757356 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:22:43.757322    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:45.711124 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:22:45.711086    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:23:05.599239 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:05.599204    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnw6b"]
Oct 30 19:23:05.599762 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:05.599597    4924 cpu_manager.go:414] "containerMap: removing container" podUID="ccddacc3-f62d-477a-9c8b-9ead25b237c4" containerName="istio-proxy"
Oct 30 19:23:05.599762 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:05.599608    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccddacc3-f62d-477a-9c8b-9ead25b237c4" containerName="istio-proxy"
Oct 30 19:23:05.599762 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:05.599621    4924 cpu_manager.go:414] "containerMap: removing container" podUID="9a6a2041-f7ce-4f30-a120-cd3b77da55a8" containerName="revoke-keys"
Oct 30 19:23:05.599762 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:05.599626    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a6a2041-f7ce-4f30-a120-cd3b77da55a8" containerName="revoke-keys"
Oct 30 19:23:05.599762 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:05.599713    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a6a2041-f7ce-4f30-a120-cd3b77da55a8" containerName="revoke-keys"
Oct 30 19:23:05.604854 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:05.604833    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnw6b"
Oct 30 19:23:05.657784 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:05.657754    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnw6b"]
Oct 30 19:23:05.702111 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:05.702058    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:23:05.702111 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:05.702079    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:23:05.702338 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:05.702057    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:23:05.702338 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:05.702152    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:23:05.740123 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:05.740055    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84fb511-1497-447c-9691-5456345073dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-pnw6b\" (UID: \"c84fb511-1497-447c-9691-5456345073dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnw6b"
Oct 30 19:23:05.740266 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:05.740208    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldww9\" (UniqueName: \"kubernetes.io/projected/c84fb511-1497-447c-9691-5456345073dc-kube-api-access-ldww9\") pod \"maas-api-revoke-sub-c1e271a9-pnw6b\" (UID: \"c84fb511-1497-447c-9691-5456345073dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnw6b"
Oct 30 19:23:05.740321 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:05.740276    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84fb511-1497-447c-9691-5456345073dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-pnw6b\" (UID: \"c84fb511-1497-447c-9691-5456345073dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnw6b"
Oct 30 19:23:05.841806 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:05.841768    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84fb511-1497-447c-9691-5456345073dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-pnw6b\" (UID: \"c84fb511-1497-447c-9691-5456345073dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnw6b"
Oct 30 19:23:05.842004 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:05.841832    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldww9\" (UniqueName: \"kubernetes.io/projected/c84fb511-1497-447c-9691-5456345073dc-kube-api-access-ldww9\") pod \"maas-api-revoke-sub-c1e271a9-pnw6b\" (UID: \"c84fb511-1497-447c-9691-5456345073dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnw6b"
Oct 30 19:23:05.842004 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:05.841879    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84fb511-1497-447c-9691-5456345073dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-pnw6b\" (UID: \"c84fb511-1497-447c-9691-5456345073dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnw6b"
Oct 30 19:23:05.842437 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:05.842415    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84fb511-1497-447c-9691-5456345073dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-pnw6b\" (UID: \"c84fb511-1497-447c-9691-5456345073dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnw6b"
Oct 30 19:23:05.844176 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:05.844156    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84fb511-1497-447c-9691-5456345073dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-pnw6b\" (UID: \"c84fb511-1497-447c-9691-5456345073dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnw6b"
Oct 30 19:23:05.896700 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:05.896645    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldww9\" (UniqueName: \"kubernetes.io/projected/c84fb511-1497-447c-9691-5456345073dc-kube-api-access-ldww9\") pod \"maas-api-revoke-sub-c1e271a9-pnw6b\" (UID: \"c84fb511-1497-447c-9691-5456345073dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnw6b"
Oct 30 19:23:05.917513 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:05.917487    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnw6b"
Oct 30 19:23:06.290969 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:06.290943    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnw6b"]
Oct 30 19:23:06.292876 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:23:06.292833    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84fb511_1497_447c_9691_5456345073dc.slice/crio-001b11cd45a42fbdc7880b362c66ab68d70d9bb4d8eaf0ae2b9e65ed57f38e00 WatchSource:0}: Error finding container 001b11cd45a42fbdc7880b362c66ab68d70d9bb4d8eaf0ae2b9e65ed57f38e00: Status 404 returned error can't find the container with id 001b11cd45a42fbdc7880b362c66ab68d70d9bb4d8eaf0ae2b9e65ed57f38e00
Oct 30 19:23:07.280734 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:07.280701    4924 generic.go:356] "Generic (PLEG): container finished" podID="c84fb511-1497-447c-9691-5456345073dc" containerID="b2fa5bde1798c5134e14921e6b9d007c724a1e4d3c1a86aabadad2ea02aa8d11" exitCode=0
Oct 30 19:23:07.281136 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:07.280791    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnw6b" event={"ID":"c84fb511-1497-447c-9691-5456345073dc","Type":"ContainerDied","Data":"b2fa5bde1798c5134e14921e6b9d007c724a1e4d3c1a86aabadad2ea02aa8d11"}
Oct 30 19:23:07.281136 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:07.280840    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnw6b" event={"ID":"c84fb511-1497-447c-9691-5456345073dc","Type":"ContainerStarted","Data":"001b11cd45a42fbdc7880b362c66ab68d70d9bb4d8eaf0ae2b9e65ed57f38e00"}
Oct 30 19:23:08.420203 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:08.420178    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnw6b"
Oct 30 19:23:08.567957 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:08.567881    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c84fb511-1497-447c-9691-5456345073dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84fb511-1497-447c-9691-5456345073dc-maas-api-openshift-ca\") pod \"c84fb511-1497-447c-9691-5456345073dc\" (UID: \"c84fb511-1497-447c-9691-5456345073dc\") "
Oct 30 19:23:08.567957 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:08.567932    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c84fb511-1497-447c-9691-5456345073dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84fb511-1497-447c-9691-5456345073dc-maas-api-xks-ca\") pod \"c84fb511-1497-447c-9691-5456345073dc\" (UID: \"c84fb511-1497-447c-9691-5456345073dc\") "
Oct 30 19:23:08.568166 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:08.567963    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c84fb511-1497-447c-9691-5456345073dc-kube-api-access-ldww9\" (UniqueName: \"kubernetes.io/projected/c84fb511-1497-447c-9691-5456345073dc-kube-api-access-ldww9\") pod \"c84fb511-1497-447c-9691-5456345073dc\" (UID: \"c84fb511-1497-447c-9691-5456345073dc\") "
Oct 30 19:23:08.568295 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:08.568272    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c84fb511-1497-447c-9691-5456345073dc-maas-api-openshift-ca" pod "c84fb511-1497-447c-9691-5456345073dc" (UID: "c84fb511-1497-447c-9691-5456345073dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:08.568390 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:08.568364    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84fb511-1497-447c-9691-5456345073dc-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:08.570129 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:08.570100    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c84fb511-1497-447c-9691-5456345073dc-maas-api-xks-ca" pod "c84fb511-1497-447c-9691-5456345073dc" (UID: "c84fb511-1497-447c-9691-5456345073dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:08.570260 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:08.570235    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c84fb511-1497-447c-9691-5456345073dc-kube-api-access-ldww9" pod "c84fb511-1497-447c-9691-5456345073dc" (UID: "c84fb511-1497-447c-9691-5456345073dc"). InnerVolumeSpecName "kube-api-access-ldww9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:08.669877 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:08.669841    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84fb511-1497-447c-9691-5456345073dc-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:08.669877 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:08.669872    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldww9\" (UniqueName: \"kubernetes.io/projected/c84fb511-1497-447c-9691-5456345073dc-kube-api-access-ldww9\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:09.293436 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:09.293405    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnw6b"
Oct 30 19:23:09.293436 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:09.293424    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnw6b" event={"ID":"c84fb511-1497-447c-9691-5456345073dc","Type":"ContainerDied","Data":"001b11cd45a42fbdc7880b362c66ab68d70d9bb4d8eaf0ae2b9e65ed57f38e00"}
Oct 30 19:23:09.293673 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:09.293478    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="001b11cd45a42fbdc7880b362c66ab68d70d9bb4d8eaf0ae2b9e65ed57f38e00"
Oct 30 19:23:15.711397 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:23:15.711360    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:23:16.909572 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:16.909540    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:17.971279 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:17.971243    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:18.286492 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:18.286387    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:37.167262 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:37.167221    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tqs5k"]
Oct 30 19:23:37.167959 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:37.167651    4924 cpu_manager.go:414] "containerMap: removing container" podUID="c84fb511-1497-447c-9691-5456345073dc" containerName="revoke-keys"
Oct 30 19:23:37.167959 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:37.167665    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c84fb511-1497-447c-9691-5456345073dc" containerName="revoke-keys"
Oct 30 19:23:37.167959 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:37.167768    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="c84fb511-1497-447c-9691-5456345073dc" containerName="revoke-keys"
Oct 30 19:23:37.170965 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:37.170943    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tqs5k"
Oct 30 19:23:37.238123 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:37.238093    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:23:37.238294 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:37.238216    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:23:37.241866 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:37.241839    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tqs5k"]
Oct 30 19:23:37.285297 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:37.285268    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:23:37.285452 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:37.285324    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:23:37.300739 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:37.300709    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg8m5\" (UniqueName: \"kubernetes.io/projected/1825e05b-4412-4cfc-bfd0-a0feb5e472fd-kube-api-access-mg8m5\") pod \"maas-api-revoke-sub-2e1f5b5f-tqs5k\" (UID: \"1825e05b-4412-4cfc-bfd0-a0feb5e472fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tqs5k"
Oct 30 19:23:37.300877 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:37.300835    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1825e05b-4412-4cfc-bfd0-a0feb5e472fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tqs5k\" (UID: \"1825e05b-4412-4cfc-bfd0-a0feb5e472fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tqs5k"
Oct 30 19:23:37.300922 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:37.300890    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1825e05b-4412-4cfc-bfd0-a0feb5e472fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tqs5k\" (UID: \"1825e05b-4412-4cfc-bfd0-a0feb5e472fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tqs5k"
Oct 30 19:23:37.401746 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:37.401715    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1825e05b-4412-4cfc-bfd0-a0feb5e472fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tqs5k\" (UID: \"1825e05b-4412-4cfc-bfd0-a0feb5e472fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tqs5k"
Oct 30 19:23:37.401908 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:37.401760    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1825e05b-4412-4cfc-bfd0-a0feb5e472fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tqs5k\" (UID: \"1825e05b-4412-4cfc-bfd0-a0feb5e472fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tqs5k"
Oct 30 19:23:37.401908 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:37.401829    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg8m5\" (UniqueName: \"kubernetes.io/projected/1825e05b-4412-4cfc-bfd0-a0feb5e472fd-kube-api-access-mg8m5\") pod \"maas-api-revoke-sub-2e1f5b5f-tqs5k\" (UID: \"1825e05b-4412-4cfc-bfd0-a0feb5e472fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tqs5k"
Oct 30 19:23:37.402469 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:37.402447    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1825e05b-4412-4cfc-bfd0-a0feb5e472fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tqs5k\" (UID: \"1825e05b-4412-4cfc-bfd0-a0feb5e472fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tqs5k"
Oct 30 19:23:37.404264 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:37.404244    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1825e05b-4412-4cfc-bfd0-a0feb5e472fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tqs5k\" (UID: \"1825e05b-4412-4cfc-bfd0-a0feb5e472fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tqs5k"
Oct 30 19:23:37.457601 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:37.457540    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg8m5\" (UniqueName: \"kubernetes.io/projected/1825e05b-4412-4cfc-bfd0-a0feb5e472fd-kube-api-access-mg8m5\") pod \"maas-api-revoke-sub-2e1f5b5f-tqs5k\" (UID: \"1825e05b-4412-4cfc-bfd0-a0feb5e472fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tqs5k"
Oct 30 19:23:37.486352 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:37.486325    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tqs5k"
Oct 30 19:23:37.854754 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:37.854730    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tqs5k"]
Oct 30 19:23:37.856580 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:23:37.856552    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1825e05b_4412_4cfc_bfd0_a0feb5e472fd.slice/crio-cd3b35610d3a23511cd18bb91aeb3d176c30bfc005c48af865ac6438719a199b WatchSource:0}: Error finding container cd3b35610d3a23511cd18bb91aeb3d176c30bfc005c48af865ac6438719a199b: Status 404 returned error can't find the container with id cd3b35610d3a23511cd18bb91aeb3d176c30bfc005c48af865ac6438719a199b
Oct 30 19:23:38.463082 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:38.462978    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tqs5k" event={"ID":"1825e05b-4412-4cfc-bfd0-a0feb5e472fd","Type":"ContainerStarted","Data":"f2f6fd31b656c51caa7f1fa620480a518c34f6c487fe5371eb95c0f54dd65ddd"}
Oct 30 19:23:38.463082 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:38.463039    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tqs5k" event={"ID":"1825e05b-4412-4cfc-bfd0-a0feb5e472fd","Type":"ContainerStarted","Data":"cd3b35610d3a23511cd18bb91aeb3d176c30bfc005c48af865ac6438719a199b"}
Oct 30 19:23:39.469219 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:39.469185    4924 generic.go:356] "Generic (PLEG): container finished" podID="1825e05b-4412-4cfc-bfd0-a0feb5e472fd" containerID="f2f6fd31b656c51caa7f1fa620480a518c34f6c487fe5371eb95c0f54dd65ddd" exitCode=0
Oct 30 19:23:39.469689 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:39.469268    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tqs5k" event={"ID":"1825e05b-4412-4cfc-bfd0-a0feb5e472fd","Type":"ContainerDied","Data":"f2f6fd31b656c51caa7f1fa620480a518c34f6c487fe5371eb95c0f54dd65ddd"}
Oct 30 19:23:40.616872 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:40.616843    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tqs5k"
Oct 30 19:23:40.737039 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:40.736961    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1825e05b-4412-4cfc-bfd0-a0feb5e472fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1825e05b-4412-4cfc-bfd0-a0feb5e472fd-maas-api-xks-ca\") pod \"1825e05b-4412-4cfc-bfd0-a0feb5e472fd\" (UID: \"1825e05b-4412-4cfc-bfd0-a0feb5e472fd\") "
Oct 30 19:23:40.737039 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:40.737028    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1825e05b-4412-4cfc-bfd0-a0feb5e472fd-kube-api-access-mg8m5\" (UniqueName: \"kubernetes.io/projected/1825e05b-4412-4cfc-bfd0-a0feb5e472fd-kube-api-access-mg8m5\") pod \"1825e05b-4412-4cfc-bfd0-a0feb5e472fd\" (UID: \"1825e05b-4412-4cfc-bfd0-a0feb5e472fd\") "
Oct 30 19:23:40.737308 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:40.737163    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1825e05b-4412-4cfc-bfd0-a0feb5e472fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1825e05b-4412-4cfc-bfd0-a0feb5e472fd-maas-api-openshift-ca\") pod \"1825e05b-4412-4cfc-bfd0-a0feb5e472fd\" (UID: \"1825e05b-4412-4cfc-bfd0-a0feb5e472fd\") "
Oct 30 19:23:40.737796 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:40.737763    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1825e05b-4412-4cfc-bfd0-a0feb5e472fd-maas-api-openshift-ca" pod "1825e05b-4412-4cfc-bfd0-a0feb5e472fd" (UID: "1825e05b-4412-4cfc-bfd0-a0feb5e472fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:40.739293 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:40.739265    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1825e05b-4412-4cfc-bfd0-a0feb5e472fd-maas-api-xks-ca" pod "1825e05b-4412-4cfc-bfd0-a0feb5e472fd" (UID: "1825e05b-4412-4cfc-bfd0-a0feb5e472fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:40.739293 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:40.739279    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1825e05b-4412-4cfc-bfd0-a0feb5e472fd-kube-api-access-mg8m5" pod "1825e05b-4412-4cfc-bfd0-a0feb5e472fd" (UID: "1825e05b-4412-4cfc-bfd0-a0feb5e472fd"). InnerVolumeSpecName "kube-api-access-mg8m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:40.838575 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:40.838537    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1825e05b-4412-4cfc-bfd0-a0feb5e472fd-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:40.838575 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:40.838576    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1825e05b-4412-4cfc-bfd0-a0feb5e472fd-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:40.838788 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:40.838594    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg8m5\" (UniqueName: \"kubernetes.io/projected/1825e05b-4412-4cfc-bfd0-a0feb5e472fd-kube-api-access-mg8m5\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:41.481769 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:41.481741    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tqs5k"
Oct 30 19:23:41.481935 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:41.481770    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tqs5k" event={"ID":"1825e05b-4412-4cfc-bfd0-a0feb5e472fd","Type":"ContainerDied","Data":"cd3b35610d3a23511cd18bb91aeb3d176c30bfc005c48af865ac6438719a199b"}
Oct 30 19:23:41.481935 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:41.481822    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd3b35610d3a23511cd18bb91aeb3d176c30bfc005c48af865ac6438719a199b"
Oct 30 19:23:41.630054 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:41.630019    4924 scope.go:122] "RemoveContainer" containerID="669404bc4632cca1cc09bbd78305d74c03759516a8e8ff235069e45e44f59acf"
Oct 30 19:23:41.638823 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:41.638801    4924 scope.go:122] "RemoveContainer" containerID="f09b3b79c9f6fe950aa23a51e89ecc97e9a0cf1b0d5deeb9c6be3187070ff89b"
Oct 30 19:23:41.646817 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:41.646801    4924 scope.go:122] "RemoveContainer" containerID="2dc671f260bd82006191b6aa46c5a218e861b55272cb7b922ba4447bd2870970"
Oct 30 19:23:41.654164 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:41.654148    4924 scope.go:122] "RemoveContainer" containerID="e12c6d9e72f9fac561cd81264c32689580bb93745f7b279980c23471eea4ecfb"
Oct 30 19:23:41.661335 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:41.661319    4924 scope.go:122] "RemoveContainer" containerID="a13d07abbe8b43479c9269b4b14c8f96d0129c626e4a664f63d2522a7c873dd5"
Oct 30 19:23:41.668858 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:41.668840    4924 scope.go:122] "RemoveContainer" containerID="4f8684a0805b765c383cb0f1d4494990c0bf44389070185beca00c85a51eb158"
Oct 30 19:23:41.676111 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:41.676094    4924 scope.go:122] "RemoveContainer" containerID="b44dd6454401bde5aa192026ec3f37ce319a8cda03bd826a19602135ed60a73d"
Oct 30 19:23:41.683130 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:41.683105    4924 scope.go:122] "RemoveContainer" containerID="d01c0ff788f2902f6fc2b616f53ee131f6a9fd426a4c3c7127539ebb25e31611"
Oct 30 19:23:44.123238 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:44.123206    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:45.710886 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:23:45.710846    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:23:49.466601 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:49.466563    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hgw5n"]
Oct 30 19:23:49.466989 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:49.466960    4924 cpu_manager.go:414] "containerMap: removing container" podUID="1825e05b-4412-4cfc-bfd0-a0feb5e472fd" containerName="revoke-keys"
Oct 30 19:23:49.466989 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:49.466972    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1825e05b-4412-4cfc-bfd0-a0feb5e472fd" containerName="revoke-keys"
Oct 30 19:23:49.467118 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:49.467078    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="1825e05b-4412-4cfc-bfd0-a0feb5e472fd" containerName="revoke-keys"
Oct 30 19:23:49.472140 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:49.472115    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hgw5n"
Oct 30 19:23:49.526093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:49.526032    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hgw5n"]
Oct 30 19:23:49.570144 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:49.570110    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:23:49.570386 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:49.570366    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:23:49.570479 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:49.570386    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:23:49.570479 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:49.570389    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:23:49.623273 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:49.623234    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d071ca-41b3-46b5-9175-881954a9fdef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-hgw5n\" (UID: \"95d071ca-41b3-46b5-9175-881954a9fdef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hgw5n"
Oct 30 19:23:49.623442 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:49.623282    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kdqg\" (UniqueName: \"kubernetes.io/projected/95d071ca-41b3-46b5-9175-881954a9fdef-kube-api-access-5kdqg\") pod \"maas-api-revoke-sub-ec843018-hgw5n\" (UID: \"95d071ca-41b3-46b5-9175-881954a9fdef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hgw5n"
Oct 30 19:23:49.623442 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:49.623346    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d071ca-41b3-46b5-9175-881954a9fdef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-hgw5n\" (UID: \"95d071ca-41b3-46b5-9175-881954a9fdef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hgw5n"
Oct 30 19:23:49.724087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:49.723946    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d071ca-41b3-46b5-9175-881954a9fdef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-hgw5n\" (UID: \"95d071ca-41b3-46b5-9175-881954a9fdef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hgw5n"
Oct 30 19:23:49.724087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:49.723987    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kdqg\" (UniqueName: \"kubernetes.io/projected/95d071ca-41b3-46b5-9175-881954a9fdef-kube-api-access-5kdqg\") pod \"maas-api-revoke-sub-ec843018-hgw5n\" (UID: \"95d071ca-41b3-46b5-9175-881954a9fdef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hgw5n"
Oct 30 19:23:49.724087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:49.724030    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d071ca-41b3-46b5-9175-881954a9fdef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-hgw5n\" (UID: \"95d071ca-41b3-46b5-9175-881954a9fdef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hgw5n"
Oct 30 19:23:49.724727 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:49.724700    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d071ca-41b3-46b5-9175-881954a9fdef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-hgw5n\" (UID: \"95d071ca-41b3-46b5-9175-881954a9fdef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hgw5n"
Oct 30 19:23:49.726386 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:49.726368    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d071ca-41b3-46b5-9175-881954a9fdef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-hgw5n\" (UID: \"95d071ca-41b3-46b5-9175-881954a9fdef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hgw5n"
Oct 30 19:23:49.778963 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:49.778932    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kdqg\" (UniqueName: \"kubernetes.io/projected/95d071ca-41b3-46b5-9175-881954a9fdef-kube-api-access-5kdqg\") pod \"maas-api-revoke-sub-ec843018-hgw5n\" (UID: \"95d071ca-41b3-46b5-9175-881954a9fdef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hgw5n"
Oct 30 19:23:49.784742 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:49.784723    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hgw5n"
Oct 30 19:23:50.154574 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:50.154149    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hgw5n"]
Oct 30 19:23:50.167155 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:23:50.167112    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d071ca_41b3_46b5_9175_881954a9fdef.slice/crio-bbedc353d65d4de31a0d6d6068f804f3f592767dc4c60907b09ccb57bc8a9568 WatchSource:0}: Error finding container bbedc353d65d4de31a0d6d6068f804f3f592767dc4c60907b09ccb57bc8a9568: Status 404 returned error can't find the container with id bbedc353d65d4de31a0d6d6068f804f3f592767dc4c60907b09ccb57bc8a9568
Oct 30 19:23:50.528046 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:50.528007    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hgw5n" event={"ID":"95d071ca-41b3-46b5-9175-881954a9fdef","Type":"ContainerStarted","Data":"4161d9f643679aea61d899728df1009d54676767b0c17d9c8b45fa03af22336f"}
Oct 30 19:23:50.528469 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:50.528095    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hgw5n" event={"ID":"95d071ca-41b3-46b5-9175-881954a9fdef","Type":"ContainerStarted","Data":"bbedc353d65d4de31a0d6d6068f804f3f592767dc4c60907b09ccb57bc8a9568"}
Oct 30 19:23:50.636765 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:50.636715    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hgw5n" podStartSLOduration=1.636701973 podStartE2EDuration="1.636701973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:50.634892347 +0000 UTC m=+2831.911942792" watchObservedRunningTime="2026-10-30 19:23:50.636701973 +0000 UTC m=+2831.913752417"
Oct 30 19:23:51.534910 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:51.534877    4924 generic.go:356] "Generic (PLEG): container finished" podID="95d071ca-41b3-46b5-9175-881954a9fdef" containerID="4161d9f643679aea61d899728df1009d54676767b0c17d9c8b45fa03af22336f" exitCode=0
Oct 30 19:23:51.535336 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:51.534973    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hgw5n" event={"ID":"95d071ca-41b3-46b5-9175-881954a9fdef","Type":"ContainerDied","Data":"4161d9f643679aea61d899728df1009d54676767b0c17d9c8b45fa03af22336f"}
Oct 30 19:23:52.676400 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:52.676371    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hgw5n"
Oct 30 19:23:52.853423 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:52.853399    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95d071ca-41b3-46b5-9175-881954a9fdef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d071ca-41b3-46b5-9175-881954a9fdef-maas-api-openshift-ca\") pod \"95d071ca-41b3-46b5-9175-881954a9fdef\" (UID: \"95d071ca-41b3-46b5-9175-881954a9fdef\") "
Oct 30 19:23:52.853627 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:52.853462    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95d071ca-41b3-46b5-9175-881954a9fdef-kube-api-access-5kdqg\" (UniqueName: \"kubernetes.io/projected/95d071ca-41b3-46b5-9175-881954a9fdef-kube-api-access-5kdqg\") pod \"95d071ca-41b3-46b5-9175-881954a9fdef\" (UID: \"95d071ca-41b3-46b5-9175-881954a9fdef\") "
Oct 30 19:23:52.853627 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:52.853504    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95d071ca-41b3-46b5-9175-881954a9fdef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d071ca-41b3-46b5-9175-881954a9fdef-maas-api-xks-ca\") pod \"95d071ca-41b3-46b5-9175-881954a9fdef\" (UID: \"95d071ca-41b3-46b5-9175-881954a9fdef\") "
Oct 30 19:23:52.853844 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:52.853819    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95d071ca-41b3-46b5-9175-881954a9fdef-maas-api-openshift-ca" pod "95d071ca-41b3-46b5-9175-881954a9fdef" (UID: "95d071ca-41b3-46b5-9175-881954a9fdef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:52.853915 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:52.853894    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d071ca-41b3-46b5-9175-881954a9fdef-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:52.855720 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:52.855696    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95d071ca-41b3-46b5-9175-881954a9fdef-maas-api-xks-ca" pod "95d071ca-41b3-46b5-9175-881954a9fdef" (UID: "95d071ca-41b3-46b5-9175-881954a9fdef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:52.855827 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:52.855804    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d071ca-41b3-46b5-9175-881954a9fdef-kube-api-access-5kdqg" pod "95d071ca-41b3-46b5-9175-881954a9fdef" (UID: "95d071ca-41b3-46b5-9175-881954a9fdef"). InnerVolumeSpecName "kube-api-access-5kdqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:52.955346 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:52.955311    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kdqg\" (UniqueName: \"kubernetes.io/projected/95d071ca-41b3-46b5-9175-881954a9fdef-kube-api-access-5kdqg\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:52.955346 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:52.955337    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d071ca-41b3-46b5-9175-881954a9fdef-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:53.546550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:53.546517    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hgw5n"
Oct 30 19:23:53.546717 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:53.546558    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hgw5n" event={"ID":"95d071ca-41b3-46b5-9175-881954a9fdef","Type":"ContainerDied","Data":"bbedc353d65d4de31a0d6d6068f804f3f592767dc4c60907b09ccb57bc8a9568"}
Oct 30 19:23:53.546717 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:23:53.546605    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbedc353d65d4de31a0d6d6068f804f3f592767dc4c60907b09ccb57bc8a9568"
Oct 30 19:24:15.711081 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:24:15.711027    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:24:24.849606 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:24.849570    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kwljn"]
Oct 30 19:24:24.849999 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:24.849965    4924 cpu_manager.go:414] "containerMap: removing container" podUID="95d071ca-41b3-46b5-9175-881954a9fdef" containerName="revoke-keys"
Oct 30 19:24:24.849999 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:24.849979    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95d071ca-41b3-46b5-9175-881954a9fdef" containerName="revoke-keys"
Oct 30 19:24:24.850100 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:24.850048    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="95d071ca-41b3-46b5-9175-881954a9fdef" containerName="revoke-keys"
Oct 30 19:24:24.854582 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:24.854561    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kwljn"
Oct 30 19:24:24.907535 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:24.907497    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kwljn"]
Oct 30 19:24:24.931426 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:24.931366    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f4b913f-25d9-492a-879c-022484c1340f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-kwljn\" (UID: \"2f4b913f-25d9-492a-879c-022484c1340f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kwljn"
Oct 30 19:24:24.931585 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:24.931488    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f4b913f-25d9-492a-879c-022484c1340f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-kwljn\" (UID: \"2f4b913f-25d9-492a-879c-022484c1340f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kwljn"
Oct 30 19:24:24.931642 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:24.931614    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4vbp\" (UniqueName: \"kubernetes.io/projected/2f4b913f-25d9-492a-879c-022484c1340f-kube-api-access-j4vbp\") pod \"maas-api-revoke-sub-046e1343-kwljn\" (UID: \"2f4b913f-25d9-492a-879c-022484c1340f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kwljn"
Oct 30 19:24:24.951387 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:24.951359    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:24:24.951387 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:24.951364    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:24:24.951562 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:24.951383    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:24:24.951562 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:24.951416    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:24:25.032553 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:25.032517    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4vbp\" (UniqueName: \"kubernetes.io/projected/2f4b913f-25d9-492a-879c-022484c1340f-kube-api-access-j4vbp\") pod \"maas-api-revoke-sub-046e1343-kwljn\" (UID: \"2f4b913f-25d9-492a-879c-022484c1340f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kwljn"
Oct 30 19:24:25.032700 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:25.032567    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f4b913f-25d9-492a-879c-022484c1340f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-kwljn\" (UID: \"2f4b913f-25d9-492a-879c-022484c1340f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kwljn"
Oct 30 19:24:25.032700 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:25.032673    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f4b913f-25d9-492a-879c-022484c1340f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-kwljn\" (UID: \"2f4b913f-25d9-492a-879c-022484c1340f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kwljn"
Oct 30 19:24:25.033345 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:25.033316    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f4b913f-25d9-492a-879c-022484c1340f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-kwljn\" (UID: \"2f4b913f-25d9-492a-879c-022484c1340f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kwljn"
Oct 30 19:24:25.034868 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:25.034842    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f4b913f-25d9-492a-879c-022484c1340f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-kwljn\" (UID: \"2f4b913f-25d9-492a-879c-022484c1340f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kwljn"
Oct 30 19:24:25.088447 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:25.088419    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4vbp\" (UniqueName: \"kubernetes.io/projected/2f4b913f-25d9-492a-879c-022484c1340f-kube-api-access-j4vbp\") pod \"maas-api-revoke-sub-046e1343-kwljn\" (UID: \"2f4b913f-25d9-492a-879c-022484c1340f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kwljn"
Oct 30 19:24:25.167105 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:25.166998    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kwljn"
Oct 30 19:24:25.539422 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:25.539396    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kwljn"]
Oct 30 19:24:25.541859 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:24:25.541821    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4b913f_25d9_492a_879c_022484c1340f.slice/crio-8401f83c895503f9e1725506ac058c5063545c5a08548abd555f8258dc45bdcd WatchSource:0}: Error finding container 8401f83c895503f9e1725506ac058c5063545c5a08548abd555f8258dc45bdcd: Status 404 returned error can't find the container with id 8401f83c895503f9e1725506ac058c5063545c5a08548abd555f8258dc45bdcd
Oct 30 19:24:25.906181 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:25.906135    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kwljn" event={"ID":"2f4b913f-25d9-492a-879c-022484c1340f","Type":"ContainerStarted","Data":"d56e9ba53ba3607600d7b14868bb25ebc6ee8dff23fcf311c7105c9589f3d428"}
Oct 30 19:24:25.906647 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:25.906199    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kwljn" event={"ID":"2f4b913f-25d9-492a-879c-022484c1340f","Type":"ContainerStarted","Data":"8401f83c895503f9e1725506ac058c5063545c5a08548abd555f8258dc45bdcd"}
Oct 30 19:24:26.028726 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:26.028668    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kwljn" podStartSLOduration=2.028652549 podStartE2EDuration="2.028652549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:26.026694601 +0000 UTC m=+2867.303745045" watchObservedRunningTime="2026-10-30 19:24:26.028652549 +0000 UTC m=+2867.305702993"
Oct 30 19:24:26.075438 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:26.075359    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b688453-qh27b"]
Oct 30 19:24:26.077098 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:26.077077    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b688453-qh27b"]
Oct 30 19:24:26.840352 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:26.840262    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90ee9990-2d90-4e92-b564-121c5080271f" path="/var/lib/kubelet/pods/90ee9990-2d90-4e92-b564-121c5080271f/volumes"
Oct 30 19:24:26.912508 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:26.912478    4924 generic.go:356] "Generic (PLEG): container finished" podID="2f4b913f-25d9-492a-879c-022484c1340f" containerID="d56e9ba53ba3607600d7b14868bb25ebc6ee8dff23fcf311c7105c9589f3d428" exitCode=0
Oct 30 19:24:26.912940 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:26.912568    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kwljn" event={"ID":"2f4b913f-25d9-492a-879c-022484c1340f","Type":"ContainerDied","Data":"d56e9ba53ba3607600d7b14868bb25ebc6ee8dff23fcf311c7105c9589f3d428"}
Oct 30 19:24:28.049848 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:28.049823    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kwljn"
Oct 30 19:24:28.164697 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:28.164666    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f4b913f-25d9-492a-879c-022484c1340f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f4b913f-25d9-492a-879c-022484c1340f-maas-api-openshift-ca\") pod \"2f4b913f-25d9-492a-879c-022484c1340f\" (UID: \"2f4b913f-25d9-492a-879c-022484c1340f\") "
Oct 30 19:24:28.164895 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:28.164755    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f4b913f-25d9-492a-879c-022484c1340f-kube-api-access-j4vbp\" (UniqueName: \"kubernetes.io/projected/2f4b913f-25d9-492a-879c-022484c1340f-kube-api-access-j4vbp\") pod \"2f4b913f-25d9-492a-879c-022484c1340f\" (UID: \"2f4b913f-25d9-492a-879c-022484c1340f\") "
Oct 30 19:24:28.164895 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:28.164840    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f4b913f-25d9-492a-879c-022484c1340f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f4b913f-25d9-492a-879c-022484c1340f-maas-api-xks-ca\") pod \"2f4b913f-25d9-492a-879c-022484c1340f\" (UID: \"2f4b913f-25d9-492a-879c-022484c1340f\") "
Oct 30 19:24:28.165408 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:28.165383    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f4b913f-25d9-492a-879c-022484c1340f-maas-api-openshift-ca" pod "2f4b913f-25d9-492a-879c-022484c1340f" (UID: "2f4b913f-25d9-492a-879c-022484c1340f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:28.167634 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:28.167600    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4b913f-25d9-492a-879c-022484c1340f-kube-api-access-j4vbp" pod "2f4b913f-25d9-492a-879c-022484c1340f" (UID: "2f4b913f-25d9-492a-879c-022484c1340f"). InnerVolumeSpecName "kube-api-access-j4vbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:28.167634 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:28.167617    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f4b913f-25d9-492a-879c-022484c1340f-maas-api-xks-ca" pod "2f4b913f-25d9-492a-879c-022484c1340f" (UID: "2f4b913f-25d9-492a-879c-022484c1340f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:28.266352 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:28.266317    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f4b913f-25d9-492a-879c-022484c1340f-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:28.266352 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:28.266347    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4vbp\" (UniqueName: \"kubernetes.io/projected/2f4b913f-25d9-492a-879c-022484c1340f-kube-api-access-j4vbp\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:28.266352 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:28.266357    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f4b913f-25d9-492a-879c-022484c1340f-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:28.924599 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:28.924571    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kwljn"
Oct 30 19:24:28.924773 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:28.924566    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kwljn" event={"ID":"2f4b913f-25d9-492a-879c-022484c1340f","Type":"ContainerDied","Data":"8401f83c895503f9e1725506ac058c5063545c5a08548abd555f8258dc45bdcd"}
Oct 30 19:24:28.924773 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:28.924692    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8401f83c895503f9e1725506ac058c5063545c5a08548abd555f8258dc45bdcd"
Oct 30 19:24:35.507970 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:35.507936    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xqtcb"]
Oct 30 19:24:35.508383 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:35.508345    4924 cpu_manager.go:414] "containerMap: removing container" podUID="2f4b913f-25d9-492a-879c-022484c1340f" containerName="revoke-keys"
Oct 30 19:24:35.508383 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:35.508357    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f4b913f-25d9-492a-879c-022484c1340f" containerName="revoke-keys"
Oct 30 19:24:35.508471 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:35.508434    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f4b913f-25d9-492a-879c-022484c1340f" containerName="revoke-keys"
Oct 30 19:24:35.511704 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:35.511681    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xqtcb"
Oct 30 19:24:35.608096 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:35.608026    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:24:35.608096 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:35.608054    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:24:35.608096 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:35.608085    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:24:35.608421 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:35.608169    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:24:35.610152 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:35.610127    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xqtcb"]
Oct 30 19:24:35.638702 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:35.638664    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822881c0-b131-4f45-82a0-22e1880f482d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-xqtcb\" (UID: \"822881c0-b131-4f45-82a0-22e1880f482d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xqtcb"
Oct 30 19:24:35.638879 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:35.638739    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822881c0-b131-4f45-82a0-22e1880f482d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-xqtcb\" (UID: \"822881c0-b131-4f45-82a0-22e1880f482d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xqtcb"
Oct 30 19:24:35.638879 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:35.638824    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x764n\" (UniqueName: \"kubernetes.io/projected/822881c0-b131-4f45-82a0-22e1880f482d-kube-api-access-x764n\") pod \"maas-api-revoke-sub-d3fba056-xqtcb\" (UID: \"822881c0-b131-4f45-82a0-22e1880f482d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xqtcb"
Oct 30 19:24:35.739433 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:35.739396    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822881c0-b131-4f45-82a0-22e1880f482d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-xqtcb\" (UID: \"822881c0-b131-4f45-82a0-22e1880f482d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xqtcb"
Oct 30 19:24:35.739433 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:35.739447    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x764n\" (UniqueName: \"kubernetes.io/projected/822881c0-b131-4f45-82a0-22e1880f482d-kube-api-access-x764n\") pod \"maas-api-revoke-sub-d3fba056-xqtcb\" (UID: \"822881c0-b131-4f45-82a0-22e1880f482d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xqtcb"
Oct 30 19:24:35.739731 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:35.739520    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822881c0-b131-4f45-82a0-22e1880f482d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-xqtcb\" (UID: \"822881c0-b131-4f45-82a0-22e1880f482d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xqtcb"
Oct 30 19:24:35.740171 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:35.740146    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822881c0-b131-4f45-82a0-22e1880f482d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-xqtcb\" (UID: \"822881c0-b131-4f45-82a0-22e1880f482d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xqtcb"
Oct 30 19:24:35.741832 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:35.741805    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822881c0-b131-4f45-82a0-22e1880f482d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-xqtcb\" (UID: \"822881c0-b131-4f45-82a0-22e1880f482d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xqtcb"
Oct 30 19:24:35.794666 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:35.794606    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x764n\" (UniqueName: \"kubernetes.io/projected/822881c0-b131-4f45-82a0-22e1880f482d-kube-api-access-x764n\") pod \"maas-api-revoke-sub-d3fba056-xqtcb\" (UID: \"822881c0-b131-4f45-82a0-22e1880f482d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xqtcb"
Oct 30 19:24:35.824450 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:35.824418    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xqtcb"
Oct 30 19:24:36.194683 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:36.194657    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xqtcb"]
Oct 30 19:24:36.198568 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:24:36.198524    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822881c0_b131_4f45_82a0_22e1880f482d.slice/crio-3d535405c34c8ab0d0957af1f98165c82c1964eb4273f65ecfc399234ebb3ab3 WatchSource:0}: Error finding container 3d535405c34c8ab0d0957af1f98165c82c1964eb4273f65ecfc399234ebb3ab3: Status 404 returned error can't find the container with id 3d535405c34c8ab0d0957af1f98165c82c1964eb4273f65ecfc399234ebb3ab3
Oct 30 19:24:36.966724 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:36.966687    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xqtcb" event={"ID":"822881c0-b131-4f45-82a0-22e1880f482d","Type":"ContainerStarted","Data":"a9f027f4671fec8212850af88aae18090387292621684068306851136d6d209d"}
Oct 30 19:24:36.967205 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:36.966758    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xqtcb" event={"ID":"822881c0-b131-4f45-82a0-22e1880f482d","Type":"ContainerStarted","Data":"3d535405c34c8ab0d0957af1f98165c82c1964eb4273f65ecfc399234ebb3ab3"}
Oct 30 19:24:37.075757 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:37.075664    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xqtcb" podStartSLOduration=2.075648009 podStartE2EDuration="2.075648009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:37.07552664 +0000 UTC m=+2878.352577098" watchObservedRunningTime="2026-10-30 19:24:37.075648009 +0000 UTC m=+2878.352698457"
Oct 30 19:24:37.976966 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:37.976921    4924 generic.go:356] "Generic (PLEG): container finished" podID="822881c0-b131-4f45-82a0-22e1880f482d" containerID="a9f027f4671fec8212850af88aae18090387292621684068306851136d6d209d" exitCode=0
Oct 30 19:24:37.977553 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:37.976983    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xqtcb" event={"ID":"822881c0-b131-4f45-82a0-22e1880f482d","Type":"ContainerDied","Data":"a9f027f4671fec8212850af88aae18090387292621684068306851136d6d209d"}
Oct 30 19:24:39.108091 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:39.103161    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb02e581-v9cws"]
Oct 30 19:24:39.125100 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:39.123242    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb02e581-v9cws"]
Oct 30 19:24:39.951093 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:39.945236    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xqtcb"
Oct 30 19:24:40.040966 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:40.040901    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xqtcb" event={"ID":"822881c0-b131-4f45-82a0-22e1880f482d","Type":"ContainerDied","Data":"3d535405c34c8ab0d0957af1f98165c82c1964eb4273f65ecfc399234ebb3ab3"}
Oct 30 19:24:40.041288 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:40.041244    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d535405c34c8ab0d0957af1f98165c82c1964eb4273f65ecfc399234ebb3ab3"
Oct 30 19:24:40.041591 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:40.041524    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xqtcb"
Oct 30 19:24:40.067331 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:40.067256    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/822881c0-b131-4f45-82a0-22e1880f482d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822881c0-b131-4f45-82a0-22e1880f482d-maas-api-openshift-ca\") pod \"822881c0-b131-4f45-82a0-22e1880f482d\" (UID: \"822881c0-b131-4f45-82a0-22e1880f482d\") "
Oct 30 19:24:40.067605 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:40.067585    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/822881c0-b131-4f45-82a0-22e1880f482d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822881c0-b131-4f45-82a0-22e1880f482d-maas-api-xks-ca\") pod \"822881c0-b131-4f45-82a0-22e1880f482d\" (UID: \"822881c0-b131-4f45-82a0-22e1880f482d\") "
Oct 30 19:24:40.067800 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:40.067784    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/822881c0-b131-4f45-82a0-22e1880f482d-kube-api-access-x764n\" (UniqueName: \"kubernetes.io/projected/822881c0-b131-4f45-82a0-22e1880f482d-kube-api-access-x764n\") pod \"822881c0-b131-4f45-82a0-22e1880f482d\" (UID: \"822881c0-b131-4f45-82a0-22e1880f482d\") "
Oct 30 19:24:40.075085 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:40.070753    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/822881c0-b131-4f45-82a0-22e1880f482d-maas-api-openshift-ca" pod "822881c0-b131-4f45-82a0-22e1880f482d" (UID: "822881c0-b131-4f45-82a0-22e1880f482d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:40.075655 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:40.075632    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822881c0-b131-4f45-82a0-22e1880f482d-kube-api-access-x764n" pod "822881c0-b131-4f45-82a0-22e1880f482d" (UID: "822881c0-b131-4f45-82a0-22e1880f482d"). InnerVolumeSpecName "kube-api-access-x764n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:40.086228 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:40.086202    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/822881c0-b131-4f45-82a0-22e1880f482d-maas-api-xks-ca" pod "822881c0-b131-4f45-82a0-22e1880f482d" (UID: "822881c0-b131-4f45-82a0-22e1880f482d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:24:40.171087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:40.170707    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x764n\" (UniqueName: \"kubernetes.io/projected/822881c0-b131-4f45-82a0-22e1880f482d-kube-api-access-x764n\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:40.171087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:40.170761    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822881c0-b131-4f45-82a0-22e1880f482d-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:40.171087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:40.170780    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822881c0-b131-4f45-82a0-22e1880f482d-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:40.833668 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:40.833640    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a242e375-9a2f-4501-bbd4-4f031a3d46f9" path="/var/lib/kubelet/pods/a242e375-9a2f-4501-bbd4-4f031a3d46f9/volumes"
Oct 30 19:24:44.443342 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:24:44.443308    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:45.711024 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:24:45.710988    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:25:07.304238 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:07.304195    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rlwcq"]
Oct 30 19:25:07.306773 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:07.304612    4924 cpu_manager.go:414] "containerMap: removing container" podUID="822881c0-b131-4f45-82a0-22e1880f482d" containerName="revoke-keys"
Oct 30 19:25:07.306773 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:07.304624    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="822881c0-b131-4f45-82a0-22e1880f482d" containerName="revoke-keys"
Oct 30 19:25:07.306773 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:07.304709    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="822881c0-b131-4f45-82a0-22e1880f482d" containerName="revoke-keys"
Oct 30 19:25:07.307773 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:07.307751    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rlwcq"
Oct 30 19:25:07.358442 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:07.358415    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:25:07.362455 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:07.362428    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e1894e-9c24-4c9a-ba79-5626c51a9baf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-rlwcq\" (UID: \"77e1894e-9c24-4c9a-ba79-5626c51a9baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rlwcq"
Oct 30 19:25:07.362605 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:07.362460    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e1894e-9c24-4c9a-ba79-5626c51a9baf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-rlwcq\" (UID: \"77e1894e-9c24-4c9a-ba79-5626c51a9baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rlwcq"
Oct 30 19:25:07.362672 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:07.362631    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7bps\" (UniqueName: \"kubernetes.io/projected/77e1894e-9c24-4c9a-ba79-5626c51a9baf-kube-api-access-c7bps\") pod \"maas-api-revoke-sub-ae594979-rlwcq\" (UID: \"77e1894e-9c24-4c9a-ba79-5626c51a9baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rlwcq"
Oct 30 19:25:07.405283 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:07.405246    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:25:07.405474 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:07.405291    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:25:07.407630 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:07.407609    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rlwcq"]
Oct 30 19:25:07.458450 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:07.458421    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:25:07.463855 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:07.463437    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e1894e-9c24-4c9a-ba79-5626c51a9baf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-rlwcq\" (UID: \"77e1894e-9c24-4c9a-ba79-5626c51a9baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rlwcq"
Oct 30 19:25:07.463855 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:07.463479    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e1894e-9c24-4c9a-ba79-5626c51a9baf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-rlwcq\" (UID: \"77e1894e-9c24-4c9a-ba79-5626c51a9baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rlwcq"
Oct 30 19:25:07.463855 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:07.463591    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7bps\" (UniqueName: \"kubernetes.io/projected/77e1894e-9c24-4c9a-ba79-5626c51a9baf-kube-api-access-c7bps\") pod \"maas-api-revoke-sub-ae594979-rlwcq\" (UID: \"77e1894e-9c24-4c9a-ba79-5626c51a9baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rlwcq"
Oct 30 19:25:07.464263 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:07.464209    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e1894e-9c24-4c9a-ba79-5626c51a9baf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-rlwcq\" (UID: \"77e1894e-9c24-4c9a-ba79-5626c51a9baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rlwcq"
Oct 30 19:25:07.466164 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:07.466145    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e1894e-9c24-4c9a-ba79-5626c51a9baf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-rlwcq\" (UID: \"77e1894e-9c24-4c9a-ba79-5626c51a9baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rlwcq"
Oct 30 19:25:07.519122 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:07.519081    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7bps\" (UniqueName: \"kubernetes.io/projected/77e1894e-9c24-4c9a-ba79-5626c51a9baf-kube-api-access-c7bps\") pod \"maas-api-revoke-sub-ae594979-rlwcq\" (UID: \"77e1894e-9c24-4c9a-ba79-5626c51a9baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rlwcq"
Oct 30 19:25:07.624282 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:07.624242    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rlwcq"
Oct 30 19:25:08.000188 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:08.000161    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rlwcq"]
Oct 30 19:25:08.002294 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:25:08.002261    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e1894e_9c24_4c9a_ba79_5626c51a9baf.slice/crio-b7ef1fea5ff99dc5c488380ded378488daccd08e8dab5c99ba09d6bdc9394ccd WatchSource:0}: Error finding container b7ef1fea5ff99dc5c488380ded378488daccd08e8dab5c99ba09d6bdc9394ccd: Status 404 returned error can't find the container with id b7ef1fea5ff99dc5c488380ded378488daccd08e8dab5c99ba09d6bdc9394ccd
Oct 30 19:25:08.258118 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:08.258016    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rlwcq" event={"ID":"77e1894e-9c24-4c9a-ba79-5626c51a9baf","Type":"ContainerStarted","Data":"b7ef1fea5ff99dc5c488380ded378488daccd08e8dab5c99ba09d6bdc9394ccd"}
Oct 30 19:25:09.264428 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:09.264340    4924 generic.go:356] "Generic (PLEG): container finished" podID="77e1894e-9c24-4c9a-ba79-5626c51a9baf" containerID="84b1b8201da1bdc5f4292e70fcdde05c1d806788a35db851faf45e8ef64589d2" exitCode=0
Oct 30 19:25:09.264818 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:09.264428    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rlwcq" event={"ID":"77e1894e-9c24-4c9a-ba79-5626c51a9baf","Type":"ContainerDied","Data":"84b1b8201da1bdc5f4292e70fcdde05c1d806788a35db851faf45e8ef64589d2"}
Oct 30 19:25:10.412164 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:10.412134    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rlwcq"
Oct 30 19:25:10.496291 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:10.496255    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77e1894e-9c24-4c9a-ba79-5626c51a9baf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e1894e-9c24-4c9a-ba79-5626c51a9baf-maas-api-xks-ca\") pod \"77e1894e-9c24-4c9a-ba79-5626c51a9baf\" (UID: \"77e1894e-9c24-4c9a-ba79-5626c51a9baf\") "
Oct 30 19:25:10.496470 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:10.496344    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77e1894e-9c24-4c9a-ba79-5626c51a9baf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e1894e-9c24-4c9a-ba79-5626c51a9baf-maas-api-openshift-ca\") pod \"77e1894e-9c24-4c9a-ba79-5626c51a9baf\" (UID: \"77e1894e-9c24-4c9a-ba79-5626c51a9baf\") "
Oct 30 19:25:10.496470 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:10.496402    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77e1894e-9c24-4c9a-ba79-5626c51a9baf-kube-api-access-c7bps\" (UniqueName: \"kubernetes.io/projected/77e1894e-9c24-4c9a-ba79-5626c51a9baf-kube-api-access-c7bps\") pod \"77e1894e-9c24-4c9a-ba79-5626c51a9baf\" (UID: \"77e1894e-9c24-4c9a-ba79-5626c51a9baf\") "
Oct 30 19:25:10.496756 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:10.496727    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77e1894e-9c24-4c9a-ba79-5626c51a9baf-maas-api-openshift-ca" pod "77e1894e-9c24-4c9a-ba79-5626c51a9baf" (UID: "77e1894e-9c24-4c9a-ba79-5626c51a9baf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:10.498510 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:10.498489    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77e1894e-9c24-4c9a-ba79-5626c51a9baf-maas-api-xks-ca" pod "77e1894e-9c24-4c9a-ba79-5626c51a9baf" (UID: "77e1894e-9c24-4c9a-ba79-5626c51a9baf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:25:10.498624 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:10.498562    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77e1894e-9c24-4c9a-ba79-5626c51a9baf-kube-api-access-c7bps" pod "77e1894e-9c24-4c9a-ba79-5626c51a9baf" (UID: "77e1894e-9c24-4c9a-ba79-5626c51a9baf"). InnerVolumeSpecName "kube-api-access-c7bps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:10.597559 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:10.597525    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e1894e-9c24-4c9a-ba79-5626c51a9baf-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:10.597559 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:10.597559    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e1894e-9c24-4c9a-ba79-5626c51a9baf-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:10.597761 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:10.597570    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7bps\" (UniqueName: \"kubernetes.io/projected/77e1894e-9c24-4c9a-ba79-5626c51a9baf-kube-api-access-c7bps\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:11.069127 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:11.069027    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:11.284339 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:11.284313    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rlwcq"
Oct 30 19:25:11.284525 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:11.284344    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rlwcq" event={"ID":"77e1894e-9c24-4c9a-ba79-5626c51a9baf","Type":"ContainerDied","Data":"b7ef1fea5ff99dc5c488380ded378488daccd08e8dab5c99ba09d6bdc9394ccd"}
Oct 30 19:25:11.284525 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:11.284397    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7ef1fea5ff99dc5c488380ded378488daccd08e8dab5c99ba09d6bdc9394ccd"
Oct 30 19:25:15.711378 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:25:15.711337    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:25:35.494793 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:35.494753    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6xlph"]
Oct 30 19:25:35.496242 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:35.496212    4924 cpu_manager.go:414] "containerMap: removing container" podUID="77e1894e-9c24-4c9a-ba79-5626c51a9baf" containerName="revoke-keys"
Oct 30 19:25:35.496242 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:35.496241    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77e1894e-9c24-4c9a-ba79-5626c51a9baf" containerName="revoke-keys"
Oct 30 19:25:35.496582 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:35.496563    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="77e1894e-9c24-4c9a-ba79-5626c51a9baf" containerName="revoke-keys"
Oct 30 19:25:35.505213 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:35.505186    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6xlph"
Oct 30 19:25:35.554705 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:35.554661    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6xlph"]
Oct 30 19:25:35.597613 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:35.597577    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd626f9-6b88-4e6d-8b31-21670e7827ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6xlph\" (UID: \"abd626f9-6b88-4e6d-8b31-21670e7827ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6xlph"
Oct 30 19:25:35.597773 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:35.597635    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd626f9-6b88-4e6d-8b31-21670e7827ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6xlph\" (UID: \"abd626f9-6b88-4e6d-8b31-21670e7827ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6xlph"
Oct 30 19:25:35.597773 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:35.597660    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9f9m\" (UniqueName: \"kubernetes.io/projected/abd626f9-6b88-4e6d-8b31-21670e7827ab-kube-api-access-v9f9m\") pod \"maas-api-revoke-sub-a4aef7d7-6xlph\" (UID: \"abd626f9-6b88-4e6d-8b31-21670e7827ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6xlph"
Oct 30 19:25:35.598529 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:35.598509    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:25:35.598693 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:35.598681    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:25:35.598745 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:35.598730    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:25:35.601428 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:35.601413    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:25:35.699283 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:35.699248    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd626f9-6b88-4e6d-8b31-21670e7827ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6xlph\" (UID: \"abd626f9-6b88-4e6d-8b31-21670e7827ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6xlph"
Oct 30 19:25:35.699492 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:35.699302    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd626f9-6b88-4e6d-8b31-21670e7827ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6xlph\" (UID: \"abd626f9-6b88-4e6d-8b31-21670e7827ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6xlph"
Oct 30 19:25:35.699492 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:35.699328    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9f9m\" (UniqueName: \"kubernetes.io/projected/abd626f9-6b88-4e6d-8b31-21670e7827ab-kube-api-access-v9f9m\") pod \"maas-api-revoke-sub-a4aef7d7-6xlph\" (UID: \"abd626f9-6b88-4e6d-8b31-21670e7827ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6xlph"
Oct 30 19:25:35.699930 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:35.699905    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd626f9-6b88-4e6d-8b31-21670e7827ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6xlph\" (UID: \"abd626f9-6b88-4e6d-8b31-21670e7827ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6xlph"
Oct 30 19:25:35.701667 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:35.701643    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd626f9-6b88-4e6d-8b31-21670e7827ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6xlph\" (UID: \"abd626f9-6b88-4e6d-8b31-21670e7827ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6xlph"
Oct 30 19:25:35.754023 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:35.753970    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9f9m\" (UniqueName: \"kubernetes.io/projected/abd626f9-6b88-4e6d-8b31-21670e7827ab-kube-api-access-v9f9m\") pod \"maas-api-revoke-sub-a4aef7d7-6xlph\" (UID: \"abd626f9-6b88-4e6d-8b31-21670e7827ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6xlph"
Oct 30 19:25:35.817318 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:35.817290    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6xlph"
Oct 30 19:25:36.189425 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:36.189401    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6xlph"]
Oct 30 19:25:36.191699 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:25:36.191655    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd626f9_6b88_4e6d_8b31_21670e7827ab.slice/crio-1e49b8c605c1c4dd9e00807a7cb081df47e6ec1c32b0d3b08061efb2eae883b6 WatchSource:0}: Error finding container 1e49b8c605c1c4dd9e00807a7cb081df47e6ec1c32b0d3b08061efb2eae883b6: Status 404 returned error can't find the container with id 1e49b8c605c1c4dd9e00807a7cb081df47e6ec1c32b0d3b08061efb2eae883b6
Oct 30 19:25:36.430991 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:36.430963    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6xlph" event={"ID":"abd626f9-6b88-4e6d-8b31-21670e7827ab","Type":"ContainerStarted","Data":"1e49b8c605c1c4dd9e00807a7cb081df47e6ec1c32b0d3b08061efb2eae883b6"}
Oct 30 19:25:37.437355 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:37.437321    4924 generic.go:356] "Generic (PLEG): container finished" podID="abd626f9-6b88-4e6d-8b31-21670e7827ab" containerID="27e0026130685afb0712d6a508addcdb02360e3cc2519d2c13d596c2b669386e" exitCode=0
Oct 30 19:25:37.437861 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:37.437410    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6xlph" event={"ID":"abd626f9-6b88-4e6d-8b31-21670e7827ab","Type":"ContainerDied","Data":"27e0026130685afb0712d6a508addcdb02360e3cc2519d2c13d596c2b669386e"}
Oct 30 19:25:38.578347 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:38.578323    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6xlph"
Oct 30 19:25:38.624368 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:38.624341    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd626f9-6b88-4e6d-8b31-21670e7827ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd626f9-6b88-4e6d-8b31-21670e7827ab-maas-api-openshift-ca\") pod \"abd626f9-6b88-4e6d-8b31-21670e7827ab\" (UID: \"abd626f9-6b88-4e6d-8b31-21670e7827ab\") "
Oct 30 19:25:38.624475 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:38.624403    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd626f9-6b88-4e6d-8b31-21670e7827ab-kube-api-access-v9f9m\" (UniqueName: \"kubernetes.io/projected/abd626f9-6b88-4e6d-8b31-21670e7827ab-kube-api-access-v9f9m\") pod \"abd626f9-6b88-4e6d-8b31-21670e7827ab\" (UID: \"abd626f9-6b88-4e6d-8b31-21670e7827ab\") "
Oct 30 19:25:38.624475 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:38.624426    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abd626f9-6b88-4e6d-8b31-21670e7827ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd626f9-6b88-4e6d-8b31-21670e7827ab-maas-api-xks-ca\") pod \"abd626f9-6b88-4e6d-8b31-21670e7827ab\" (UID: \"abd626f9-6b88-4e6d-8b31-21670e7827ab\") "
Oct 30 19:25:38.624712 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:38.624687    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd626f9-6b88-4e6d-8b31-21670e7827ab-maas-api-openshift-ca" pod "abd626f9-6b88-4e6d-8b31-21670e7827ab" (UID: "abd626f9-6b88-4e6d-8b31-21670e7827ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:38.624800 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:38.624786    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd626f9-6b88-4e6d-8b31-21670e7827ab-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:38.626423 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:38.626401    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd626f9-6b88-4e6d-8b31-21670e7827ab-kube-api-access-v9f9m" pod "abd626f9-6b88-4e6d-8b31-21670e7827ab" (UID: "abd626f9-6b88-4e6d-8b31-21670e7827ab"). InnerVolumeSpecName "kube-api-access-v9f9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:38.626537 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:38.626496    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abd626f9-6b88-4e6d-8b31-21670e7827ab-maas-api-xks-ca" pod "abd626f9-6b88-4e6d-8b31-21670e7827ab" (UID: "abd626f9-6b88-4e6d-8b31-21670e7827ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:25:38.725637 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:38.725561    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9f9m\" (UniqueName: \"kubernetes.io/projected/abd626f9-6b88-4e6d-8b31-21670e7827ab-kube-api-access-v9f9m\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:38.725637 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:38.725590    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd626f9-6b88-4e6d-8b31-21670e7827ab-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:39.448520 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:39.448487    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6xlph"
Oct 30 19:25:39.448520 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:39.448509    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6xlph" event={"ID":"abd626f9-6b88-4e6d-8b31-21670e7827ab","Type":"ContainerDied","Data":"1e49b8c605c1c4dd9e00807a7cb081df47e6ec1c32b0d3b08061efb2eae883b6"}
Oct 30 19:25:39.448772 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:39.448554    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e49b8c605c1c4dd9e00807a7cb081df47e6ec1c32b0d3b08061efb2eae883b6"
Oct 30 19:25:41.734822 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:41.734789    4924 scope.go:122] "RemoveContainer" containerID="999ad7e52f34bfbb51d887f700c2d17cb79e7a1ea55e2e7c9e4ff45b9f898cd9"
Oct 30 19:25:41.743640 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:41.743622    4924 scope.go:122] "RemoveContainer" containerID="4f68f0b9d5840f140843d4e96922a9f2916d21c2c81217ca22334d7d83a9269f"
Oct 30 19:25:44.784702 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:25:44.784673    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:45.709948 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:25:45.709912    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:26:05.787318 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:05.787277    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jk7f9"]
Oct 30 19:26:05.787864 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:05.787661    4924 cpu_manager.go:414] "containerMap: removing container" podUID="abd626f9-6b88-4e6d-8b31-21670e7827ab" containerName="revoke-keys"
Oct 30 19:26:05.787864 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:05.787675    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd626f9-6b88-4e6d-8b31-21670e7827ab" containerName="revoke-keys"
Oct 30 19:26:05.787864 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:05.787765    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd626f9-6b88-4e6d-8b31-21670e7827ab" containerName="revoke-keys"
Oct 30 19:26:05.792248 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:05.792226    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jk7f9"
Oct 30 19:26:05.888630 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:05.888577    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:26:05.888630 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:05.888612    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:26:05.888896 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:05.888807    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:26:05.891261 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:05.891244    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jk7f9"]
Oct 30 19:26:05.919367 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:05.919346    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dee08af-8e31-40cc-8913-99f389aeb86c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-jk7f9\" (UID: \"8dee08af-8e31-40cc-8913-99f389aeb86c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jk7f9"
Oct 30 19:26:05.919569 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:05.919549    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs6bs\" (UniqueName: \"kubernetes.io/projected/8dee08af-8e31-40cc-8913-99f389aeb86c-kube-api-access-vs6bs\") pod \"maas-api-revoke-sub-79805ca9-jk7f9\" (UID: \"8dee08af-8e31-40cc-8913-99f389aeb86c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jk7f9"
Oct 30 19:26:05.919652 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:05.919588    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dee08af-8e31-40cc-8913-99f389aeb86c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-jk7f9\" (UID: \"8dee08af-8e31-40cc-8913-99f389aeb86c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jk7f9"
Oct 30 19:26:05.943289 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:05.943271    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:26:06.020451 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:06.020426    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs6bs\" (UniqueName: \"kubernetes.io/projected/8dee08af-8e31-40cc-8913-99f389aeb86c-kube-api-access-vs6bs\") pod \"maas-api-revoke-sub-79805ca9-jk7f9\" (UID: \"8dee08af-8e31-40cc-8913-99f389aeb86c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jk7f9"
Oct 30 19:26:06.020581 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:06.020456    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dee08af-8e31-40cc-8913-99f389aeb86c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-jk7f9\" (UID: \"8dee08af-8e31-40cc-8913-99f389aeb86c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jk7f9"
Oct 30 19:26:06.020581 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:06.020481    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dee08af-8e31-40cc-8913-99f389aeb86c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-jk7f9\" (UID: \"8dee08af-8e31-40cc-8913-99f389aeb86c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jk7f9"
Oct 30 19:26:06.021102 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:06.021081    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dee08af-8e31-40cc-8913-99f389aeb86c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-jk7f9\" (UID: \"8dee08af-8e31-40cc-8913-99f389aeb86c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jk7f9"
Oct 30 19:26:06.022747 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:06.022725    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dee08af-8e31-40cc-8913-99f389aeb86c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-jk7f9\" (UID: \"8dee08af-8e31-40cc-8913-99f389aeb86c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jk7f9"
Oct 30 19:26:06.076274 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:06.076216    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs6bs\" (UniqueName: \"kubernetes.io/projected/8dee08af-8e31-40cc-8913-99f389aeb86c-kube-api-access-vs6bs\") pod \"maas-api-revoke-sub-79805ca9-jk7f9\" (UID: \"8dee08af-8e31-40cc-8913-99f389aeb86c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jk7f9"
Oct 30 19:26:06.104888 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:06.104859    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jk7f9"
Oct 30 19:26:06.470772 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:06.470744    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jk7f9"]
Oct 30 19:26:06.473001 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:26:06.472967    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dee08af_8e31_40cc_8913_99f389aeb86c.slice/crio-55e8a290b58fc3bb1c1249d814498ef89c11b46d0a4ef86c78bbe5cd72402d8f WatchSource:0}: Error finding container 55e8a290b58fc3bb1c1249d814498ef89c11b46d0a4ef86c78bbe5cd72402d8f: Status 404 returned error can't find the container with id 55e8a290b58fc3bb1c1249d814498ef89c11b46d0a4ef86c78bbe5cd72402d8f
Oct 30 19:26:06.609575 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:06.609543    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jk7f9" event={"ID":"8dee08af-8e31-40cc-8913-99f389aeb86c","Type":"ContainerStarted","Data":"55e8a290b58fc3bb1c1249d814498ef89c11b46d0a4ef86c78bbe5cd72402d8f"}
Oct 30 19:26:07.617563 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:07.617529    4924 generic.go:356] "Generic (PLEG): container finished" podID="8dee08af-8e31-40cc-8913-99f389aeb86c" containerID="d1ea99f5f29039a740b7bb4b4277b487fa77474303d6a3b12e2b9a78672bd4cc" exitCode=0
Oct 30 19:26:07.617982 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:07.617570    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jk7f9" event={"ID":"8dee08af-8e31-40cc-8913-99f389aeb86c","Type":"ContainerDied","Data":"d1ea99f5f29039a740b7bb4b4277b487fa77474303d6a3b12e2b9a78672bd4cc"}
Oct 30 19:26:08.755531 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:08.755505    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jk7f9"
Oct 30 19:26:08.847054 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:08.847025    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dee08af-8e31-40cc-8913-99f389aeb86c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dee08af-8e31-40cc-8913-99f389aeb86c-maas-api-openshift-ca\") pod \"8dee08af-8e31-40cc-8913-99f389aeb86c\" (UID: \"8dee08af-8e31-40cc-8913-99f389aeb86c\") "
Oct 30 19:26:08.847226 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:08.847116    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dee08af-8e31-40cc-8913-99f389aeb86c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dee08af-8e31-40cc-8913-99f389aeb86c-maas-api-xks-ca\") pod \"8dee08af-8e31-40cc-8913-99f389aeb86c\" (UID: \"8dee08af-8e31-40cc-8913-99f389aeb86c\") "
Oct 30 19:26:08.847226 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:08.847184    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dee08af-8e31-40cc-8913-99f389aeb86c-kube-api-access-vs6bs\" (UniqueName: \"kubernetes.io/projected/8dee08af-8e31-40cc-8913-99f389aeb86c-kube-api-access-vs6bs\") pod \"8dee08af-8e31-40cc-8913-99f389aeb86c\" (UID: \"8dee08af-8e31-40cc-8913-99f389aeb86c\") "
Oct 30 19:26:08.847468 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:08.847426    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dee08af-8e31-40cc-8913-99f389aeb86c-maas-api-openshift-ca" pod "8dee08af-8e31-40cc-8913-99f389aeb86c" (UID: "8dee08af-8e31-40cc-8913-99f389aeb86c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:08.849281 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:08.849257    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dee08af-8e31-40cc-8913-99f389aeb86c-maas-api-xks-ca" pod "8dee08af-8e31-40cc-8913-99f389aeb86c" (UID: "8dee08af-8e31-40cc-8913-99f389aeb86c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:26:08.849362 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:08.849298    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dee08af-8e31-40cc-8913-99f389aeb86c-kube-api-access-vs6bs" pod "8dee08af-8e31-40cc-8913-99f389aeb86c" (UID: "8dee08af-8e31-40cc-8913-99f389aeb86c"). InnerVolumeSpecName "kube-api-access-vs6bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:08.948747 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:08.948667    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dee08af-8e31-40cc-8913-99f389aeb86c-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:08.948747 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:08.948698    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dee08af-8e31-40cc-8913-99f389aeb86c-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:08.948747 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:08.948708    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs6bs\" (UniqueName: \"kubernetes.io/projected/8dee08af-8e31-40cc-8913-99f389aeb86c-kube-api-access-vs6bs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:09.628541 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:09.628507    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jk7f9"
Oct 30 19:26:09.628813 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:09.628536    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jk7f9" event={"ID":"8dee08af-8e31-40cc-8913-99f389aeb86c","Type":"ContainerDied","Data":"55e8a290b58fc3bb1c1249d814498ef89c11b46d0a4ef86c78bbe5cd72402d8f"}
Oct 30 19:26:09.628813 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:09.628586    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55e8a290b58fc3bb1c1249d814498ef89c11b46d0a4ef86c78bbe5cd72402d8f"
Oct 30 19:26:14.048559 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:14.048521    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba7f230-llzws"]
Oct 30 19:26:14.095329 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:14.095295    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba7f230-llzws"]
Oct 30 19:26:14.833335 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:14.833301    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29a1972f-7378-4c07-9957-ca1ca5e854e3" path="/var/lib/kubelet/pods/29a1972f-7378-4c07-9957-ca1ca5e854e3/volumes"
Oct 30 19:26:15.710153 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:26:15.710118    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:26:26.421759 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:26.421721    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f7g99"]
Oct 30 19:26:26.422205 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:26.422195    4924 cpu_manager.go:414] "containerMap: removing container" podUID="8dee08af-8e31-40cc-8913-99f389aeb86c" containerName="revoke-keys"
Oct 30 19:26:26.422266 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:26.422210    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dee08af-8e31-40cc-8913-99f389aeb86c" containerName="revoke-keys"
Oct 30 19:26:26.422323 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:26.422312    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dee08af-8e31-40cc-8913-99f389aeb86c" containerName="revoke-keys"
Oct 30 19:26:26.425760 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:26.425737    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f7g99"
Oct 30 19:26:26.480209 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:26.480177    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f7g99"]
Oct 30 19:26:26.522696 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:26.522670    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecbc2b45-09db-4770-8490-af7a15ccac8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-f7g99\" (UID: \"ecbc2b45-09db-4770-8490-af7a15ccac8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f7g99"
Oct 30 19:26:26.522867 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:26.522702    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecbc2b45-09db-4770-8490-af7a15ccac8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-f7g99\" (UID: \"ecbc2b45-09db-4770-8490-af7a15ccac8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f7g99"
Oct 30 19:26:26.522867 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:26.522778    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfgd6\" (UniqueName: \"kubernetes.io/projected/ecbc2b45-09db-4770-8490-af7a15ccac8c-kube-api-access-cfgd6\") pod \"maas-api-revoke-sub-0b7e8edb-f7g99\" (UID: \"ecbc2b45-09db-4770-8490-af7a15ccac8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f7g99"
Oct 30 19:26:26.524142 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:26.524124    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:26:26.524142 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:26.524135    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:26:26.524263 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:26.524135    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:26:26.524317 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:26.524304    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:26:26.623448 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:26.623408    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecbc2b45-09db-4770-8490-af7a15ccac8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-f7g99\" (UID: \"ecbc2b45-09db-4770-8490-af7a15ccac8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f7g99"
Oct 30 19:26:26.623448 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:26.623447    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecbc2b45-09db-4770-8490-af7a15ccac8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-f7g99\" (UID: \"ecbc2b45-09db-4770-8490-af7a15ccac8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f7g99"
Oct 30 19:26:26.623723 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:26.623474    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfgd6\" (UniqueName: \"kubernetes.io/projected/ecbc2b45-09db-4770-8490-af7a15ccac8c-kube-api-access-cfgd6\") pod \"maas-api-revoke-sub-0b7e8edb-f7g99\" (UID: \"ecbc2b45-09db-4770-8490-af7a15ccac8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f7g99"
Oct 30 19:26:26.624180 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:26.624152    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecbc2b45-09db-4770-8490-af7a15ccac8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-f7g99\" (UID: \"ecbc2b45-09db-4770-8490-af7a15ccac8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f7g99"
Oct 30 19:26:26.625883 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:26.625863    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecbc2b45-09db-4770-8490-af7a15ccac8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-f7g99\" (UID: \"ecbc2b45-09db-4770-8490-af7a15ccac8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f7g99"
Oct 30 19:26:26.679147 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:26.679082    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfgd6\" (UniqueName: \"kubernetes.io/projected/ecbc2b45-09db-4770-8490-af7a15ccac8c-kube-api-access-cfgd6\") pod \"maas-api-revoke-sub-0b7e8edb-f7g99\" (UID: \"ecbc2b45-09db-4770-8490-af7a15ccac8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f7g99"
Oct 30 19:26:26.738295 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:26.738263    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f7g99"
Oct 30 19:26:27.102859 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:27.102829    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f7g99"]
Oct 30 19:26:27.104338 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:26:27.104311    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecbc2b45_09db_4770_8490_af7a15ccac8c.slice/crio-e905e639fd40cce960c3483315390e12e24c4f62c927fa3b51318ee2707d20b9 WatchSource:0}: Error finding container e905e639fd40cce960c3483315390e12e24c4f62c927fa3b51318ee2707d20b9: Status 404 returned error can't find the container with id e905e639fd40cce960c3483315390e12e24c4f62c927fa3b51318ee2707d20b9
Oct 30 19:26:27.755137 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:27.755039    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f7g99" event={"ID":"ecbc2b45-09db-4770-8490-af7a15ccac8c","Type":"ContainerStarted","Data":"cd02db617cb3a4653872c2885f2fac88f6973b313e45ba5e9f202512178326df"}
Oct 30 19:26:27.755137 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:27.755119    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f7g99" event={"ID":"ecbc2b45-09db-4770-8490-af7a15ccac8c","Type":"ContainerStarted","Data":"e905e639fd40cce960c3483315390e12e24c4f62c927fa3b51318ee2707d20b9"}
Oct 30 19:26:27.867415 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:27.867358    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f7g99" podStartSLOduration=1.867340473 podStartE2EDuration="1.867340473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:27.864838506 +0000 UTC m=+2989.141888956" watchObservedRunningTime="2026-10-30 19:26:27.867340473 +0000 UTC m=+2989.144390912"
Oct 30 19:26:28.761923 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:28.761890    4924 generic.go:356] "Generic (PLEG): container finished" podID="ecbc2b45-09db-4770-8490-af7a15ccac8c" containerID="cd02db617cb3a4653872c2885f2fac88f6973b313e45ba5e9f202512178326df" exitCode=0
Oct 30 19:26:28.762339 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:28.761975    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f7g99" event={"ID":"ecbc2b45-09db-4770-8490-af7a15ccac8c","Type":"ContainerDied","Data":"cd02db617cb3a4653872c2885f2fac88f6973b313e45ba5e9f202512178326df"}
Oct 30 19:26:29.910146 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:29.910123    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f7g99"
Oct 30 19:26:29.951880 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:29.951857    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecbc2b45-09db-4770-8490-af7a15ccac8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecbc2b45-09db-4770-8490-af7a15ccac8c-maas-api-xks-ca\") pod \"ecbc2b45-09db-4770-8490-af7a15ccac8c\" (UID: \"ecbc2b45-09db-4770-8490-af7a15ccac8c\") "
Oct 30 19:26:29.952011 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:29.951930    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecbc2b45-09db-4770-8490-af7a15ccac8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecbc2b45-09db-4770-8490-af7a15ccac8c-maas-api-openshift-ca\") pod \"ecbc2b45-09db-4770-8490-af7a15ccac8c\" (UID: \"ecbc2b45-09db-4770-8490-af7a15ccac8c\") "
Oct 30 19:26:29.952011 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:29.951981    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecbc2b45-09db-4770-8490-af7a15ccac8c-kube-api-access-cfgd6\" (UniqueName: \"kubernetes.io/projected/ecbc2b45-09db-4770-8490-af7a15ccac8c-kube-api-access-cfgd6\") pod \"ecbc2b45-09db-4770-8490-af7a15ccac8c\" (UID: \"ecbc2b45-09db-4770-8490-af7a15ccac8c\") "
Oct 30 19:26:29.952464 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:29.952439    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecbc2b45-09db-4770-8490-af7a15ccac8c-maas-api-openshift-ca" pod "ecbc2b45-09db-4770-8490-af7a15ccac8c" (UID: "ecbc2b45-09db-4770-8490-af7a15ccac8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:29.953894 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:29.953873    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecbc2b45-09db-4770-8490-af7a15ccac8c-maas-api-xks-ca" pod "ecbc2b45-09db-4770-8490-af7a15ccac8c" (UID: "ecbc2b45-09db-4770-8490-af7a15ccac8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:26:29.954011 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:29.953990    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecbc2b45-09db-4770-8490-af7a15ccac8c-kube-api-access-cfgd6" pod "ecbc2b45-09db-4770-8490-af7a15ccac8c" (UID: "ecbc2b45-09db-4770-8490-af7a15ccac8c"). InnerVolumeSpecName "kube-api-access-cfgd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:30.053452 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:30.053363    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecbc2b45-09db-4770-8490-af7a15ccac8c-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:30.053452 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:30.053389    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecbc2b45-09db-4770-8490-af7a15ccac8c-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:30.053452 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:30.053399    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfgd6\" (UniqueName: \"kubernetes.io/projected/ecbc2b45-09db-4770-8490-af7a15ccac8c-kube-api-access-cfgd6\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:30.779711 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:30.779675    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f7g99"
Oct 30 19:26:30.779711 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:30.779702    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f7g99" event={"ID":"ecbc2b45-09db-4770-8490-af7a15ccac8c","Type":"ContainerDied","Data":"e905e639fd40cce960c3483315390e12e24c4f62c927fa3b51318ee2707d20b9"}
Oct 30 19:26:30.779984 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:30.779773    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e905e639fd40cce960c3483315390e12e24c4f62c927fa3b51318ee2707d20b9"
Oct 30 19:26:37.701221 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:37.701188    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8ctz"]
Oct 30 19:26:37.701676 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:37.701616    4924 cpu_manager.go:414] "containerMap: removing container" podUID="ecbc2b45-09db-4770-8490-af7a15ccac8c" containerName="revoke-keys"
Oct 30 19:26:37.701676 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:37.701629    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecbc2b45-09db-4770-8490-af7a15ccac8c" containerName="revoke-keys"
Oct 30 19:26:37.701773 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:37.701727    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecbc2b45-09db-4770-8490-af7a15ccac8c" containerName="revoke-keys"
Oct 30 19:26:37.705276 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:37.705252    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8ctz"
Oct 30 19:26:37.710169 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:37.710143    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb6rx\" (UniqueName: \"kubernetes.io/projected/92181cc3-83a9-49ee-b61f-892891560df1-kube-api-access-hb6rx\") pod \"maas-api-revoke-sub-3d4156f1-v8ctz\" (UID: \"92181cc3-83a9-49ee-b61f-892891560df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8ctz"
Oct 30 19:26:37.710302 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:37.710218    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92181cc3-83a9-49ee-b61f-892891560df1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-v8ctz\" (UID: \"92181cc3-83a9-49ee-b61f-892891560df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8ctz"
Oct 30 19:26:37.710302 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:37.710284    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92181cc3-83a9-49ee-b61f-892891560df1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-v8ctz\" (UID: \"92181cc3-83a9-49ee-b61f-892891560df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8ctz"
Oct 30 19:26:37.760279 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:37.760246    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8ctz"]
Oct 30 19:26:37.803844 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:37.803814    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:26:37.804075 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:37.804048    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:26:37.804125 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:37.804112    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:26:37.804211 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:37.804192    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:26:37.810658 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:37.810638    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92181cc3-83a9-49ee-b61f-892891560df1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-v8ctz\" (UID: \"92181cc3-83a9-49ee-b61f-892891560df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8ctz"
Oct 30 19:26:37.810750 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:37.810681    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb6rx\" (UniqueName: \"kubernetes.io/projected/92181cc3-83a9-49ee-b61f-892891560df1-kube-api-access-hb6rx\") pod \"maas-api-revoke-sub-3d4156f1-v8ctz\" (UID: \"92181cc3-83a9-49ee-b61f-892891560df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8ctz"
Oct 30 19:26:37.810750 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:37.810731    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92181cc3-83a9-49ee-b61f-892891560df1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-v8ctz\" (UID: \"92181cc3-83a9-49ee-b61f-892891560df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8ctz"
Oct 30 19:26:37.811462 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:37.811442    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92181cc3-83a9-49ee-b61f-892891560df1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-v8ctz\" (UID: \"92181cc3-83a9-49ee-b61f-892891560df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8ctz"
Oct 30 19:26:37.813271 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:37.813248    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92181cc3-83a9-49ee-b61f-892891560df1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-v8ctz\" (UID: \"92181cc3-83a9-49ee-b61f-892891560df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8ctz"
Oct 30 19:26:37.865785 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:37.865752    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb6rx\" (UniqueName: \"kubernetes.io/projected/92181cc3-83a9-49ee-b61f-892891560df1-kube-api-access-hb6rx\") pod \"maas-api-revoke-sub-3d4156f1-v8ctz\" (UID: \"92181cc3-83a9-49ee-b61f-892891560df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8ctz"
Oct 30 19:26:38.018303 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:38.018219    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8ctz"
Oct 30 19:26:38.380983 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:38.380948    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8ctz"]
Oct 30 19:26:38.382586 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:26:38.382561    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92181cc3_83a9_49ee_b61f_892891560df1.slice/crio-c9a31f359eefea1a3d12b8d2b4b844b7a347bae33e880850ba8f087a32339689 WatchSource:0}: Error finding container c9a31f359eefea1a3d12b8d2b4b844b7a347bae33e880850ba8f087a32339689: Status 404 returned error can't find the container with id c9a31f359eefea1a3d12b8d2b4b844b7a347bae33e880850ba8f087a32339689
Oct 30 19:26:38.823992 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:38.823957    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8ctz" event={"ID":"92181cc3-83a9-49ee-b61f-892891560df1","Type":"ContainerStarted","Data":"f0380d702a1adf2a45e5e5cd3a7b663a3e53c147a8fa3fe73a315a29716ae5d4"}
Oct 30 19:26:38.824406 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:38.824029    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8ctz" event={"ID":"92181cc3-83a9-49ee-b61f-892891560df1","Type":"ContainerStarted","Data":"c9a31f359eefea1a3d12b8d2b4b844b7a347bae33e880850ba8f087a32339689"}
Oct 30 19:26:38.932337 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:38.932291    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8ctz" podStartSLOduration=1.932276269 podStartE2EDuration="1.932276269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:38.931239661 +0000 UTC m=+3000.208290106" watchObservedRunningTime="2026-10-30 19:26:38.932276269 +0000 UTC m=+3000.209326713"
Oct 30 19:26:39.350884 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:39.350853    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 19:26:39.352342 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:39.352320    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 19:26:39.844194 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:39.844157    4924 generic.go:356] "Generic (PLEG): container finished" podID="92181cc3-83a9-49ee-b61f-892891560df1" containerID="f0380d702a1adf2a45e5e5cd3a7b663a3e53c147a8fa3fe73a315a29716ae5d4" exitCode=0
Oct 30 19:26:39.844918 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:39.844889    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8ctz" event={"ID":"92181cc3-83a9-49ee-b61f-892891560df1","Type":"ContainerDied","Data":"f0380d702a1adf2a45e5e5cd3a7b663a3e53c147a8fa3fe73a315a29716ae5d4"}
Oct 30 19:26:41.471940 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:41.471905    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8ctz"
Oct 30 19:26:41.568718 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:41.568622    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92181cc3-83a9-49ee-b61f-892891560df1-kube-api-access-hb6rx\" (UniqueName: \"kubernetes.io/projected/92181cc3-83a9-49ee-b61f-892891560df1-kube-api-access-hb6rx\") pod \"92181cc3-83a9-49ee-b61f-892891560df1\" (UID: \"92181cc3-83a9-49ee-b61f-892891560df1\") "
Oct 30 19:26:41.568718 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:41.568681    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92181cc3-83a9-49ee-b61f-892891560df1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92181cc3-83a9-49ee-b61f-892891560df1-maas-api-xks-ca\") pod \"92181cc3-83a9-49ee-b61f-892891560df1\" (UID: \"92181cc3-83a9-49ee-b61f-892891560df1\") "
Oct 30 19:26:41.568965 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:41.568767    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92181cc3-83a9-49ee-b61f-892891560df1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92181cc3-83a9-49ee-b61f-892891560df1-maas-api-openshift-ca\") pod \"92181cc3-83a9-49ee-b61f-892891560df1\" (UID: \"92181cc3-83a9-49ee-b61f-892891560df1\") "
Oct 30 19:26:41.569173 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:41.569148    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92181cc3-83a9-49ee-b61f-892891560df1-maas-api-openshift-ca" pod "92181cc3-83a9-49ee-b61f-892891560df1" (UID: "92181cc3-83a9-49ee-b61f-892891560df1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:41.570777 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:41.570746    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92181cc3-83a9-49ee-b61f-892891560df1-maas-api-xks-ca" pod "92181cc3-83a9-49ee-b61f-892891560df1" (UID: "92181cc3-83a9-49ee-b61f-892891560df1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:26:41.570777 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:41.570750    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92181cc3-83a9-49ee-b61f-892891560df1-kube-api-access-hb6rx" pod "92181cc3-83a9-49ee-b61f-892891560df1" (UID: "92181cc3-83a9-49ee-b61f-892891560df1"). InnerVolumeSpecName "kube-api-access-hb6rx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:41.669822 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:41.669792    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92181cc3-83a9-49ee-b61f-892891560df1-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:41.669822 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:41.669820    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92181cc3-83a9-49ee-b61f-892891560df1-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:41.669822 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:41.669829    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb6rx\" (UniqueName: \"kubernetes.io/projected/92181cc3-83a9-49ee-b61f-892891560df1-kube-api-access-hb6rx\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:41.767553 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:41.767523    4924 scope.go:122] "RemoveContainer" containerID="12cdb6dfe0d9dda3c707d4a81aebb6392c5d1a5863b429e046b2fbc627b0e00a"
Oct 30 19:26:41.776974 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:41.776953    4924 scope.go:122] "RemoveContainer" containerID="82e42c2f6a1e2e117f5d92fd07e6589b8d649c71c28d310c24de75492eef80ef"
Oct 30 19:26:41.885631 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:41.885595    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8ctz"
Oct 30 19:26:41.885812 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:41.885647    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8ctz" event={"ID":"92181cc3-83a9-49ee-b61f-892891560df1","Type":"ContainerDied","Data":"c9a31f359eefea1a3d12b8d2b4b844b7a347bae33e880850ba8f087a32339689"}
Oct 30 19:26:41.885812 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:41.885690    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9a31f359eefea1a3d12b8d2b4b844b7a347bae33e880850ba8f087a32339689"
Oct 30 19:26:45.114043 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:45.114012    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:45.520328 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:26:45.520242    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:45.710801 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:26:45.710764    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:27:08.160620 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:08.160536    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8ctz"]
Oct 30 19:27:08.207252 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:08.207210    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-v8ctz"]
Oct 30 19:27:08.282547 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:08.282509    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2mmq"]
Oct 30 19:27:08.282985 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:08.282966    4924 cpu_manager.go:414] "containerMap: removing container" podUID="92181cc3-83a9-49ee-b61f-892891560df1" containerName="revoke-keys"
Oct 30 19:27:08.283104 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:08.282988    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92181cc3-83a9-49ee-b61f-892891560df1" containerName="revoke-keys"
Oct 30 19:27:08.283177 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:08.283148    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="92181cc3-83a9-49ee-b61f-892891560df1" containerName="revoke-keys"
Oct 30 19:27:08.288028 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:08.288007    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2mmq"
Oct 30 19:27:08.340653 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:08.340625    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:27:08.340802 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:08.340695    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:27:08.340926 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:08.340910    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:27:08.342876 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:08.342846    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2mmq"]
Oct 30 19:27:08.365387 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:08.365359    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3dd5295-ae7b-40e0-ae62-ae44c14fbf90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p2mmq\" (UID: \"d3dd5295-ae7b-40e0-ae62-ae44c14fbf90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2mmq"
Oct 30 19:27:08.365570 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:08.365504    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3dd5295-ae7b-40e0-ae62-ae44c14fbf90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p2mmq\" (UID: \"d3dd5295-ae7b-40e0-ae62-ae44c14fbf90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2mmq"
Oct 30 19:27:08.365570 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:08.365554    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9p8n\" (UniqueName: \"kubernetes.io/projected/d3dd5295-ae7b-40e0-ae62-ae44c14fbf90-kube-api-access-t9p8n\") pod \"maas-api-revoke-sub-3d4156f1-p2mmq\" (UID: \"d3dd5295-ae7b-40e0-ae62-ae44c14fbf90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2mmq"
Oct 30 19:27:08.386756 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:08.386726    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:27:08.466722 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:08.466631    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9p8n\" (UniqueName: \"kubernetes.io/projected/d3dd5295-ae7b-40e0-ae62-ae44c14fbf90-kube-api-access-t9p8n\") pod \"maas-api-revoke-sub-3d4156f1-p2mmq\" (UID: \"d3dd5295-ae7b-40e0-ae62-ae44c14fbf90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2mmq"
Oct 30 19:27:08.466722 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:08.466686    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3dd5295-ae7b-40e0-ae62-ae44c14fbf90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p2mmq\" (UID: \"d3dd5295-ae7b-40e0-ae62-ae44c14fbf90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2mmq"
Oct 30 19:27:08.466983 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:08.466745    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3dd5295-ae7b-40e0-ae62-ae44c14fbf90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p2mmq\" (UID: \"d3dd5295-ae7b-40e0-ae62-ae44c14fbf90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2mmq"
Oct 30 19:27:08.467429 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:08.467408    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3dd5295-ae7b-40e0-ae62-ae44c14fbf90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p2mmq\" (UID: \"d3dd5295-ae7b-40e0-ae62-ae44c14fbf90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2mmq"
Oct 30 19:27:08.469190 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:08.469169    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3dd5295-ae7b-40e0-ae62-ae44c14fbf90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p2mmq\" (UID: \"d3dd5295-ae7b-40e0-ae62-ae44c14fbf90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2mmq"
Oct 30 19:27:08.521549 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:08.521515    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9p8n\" (UniqueName: \"kubernetes.io/projected/d3dd5295-ae7b-40e0-ae62-ae44c14fbf90-kube-api-access-t9p8n\") pod \"maas-api-revoke-sub-3d4156f1-p2mmq\" (UID: \"d3dd5295-ae7b-40e0-ae62-ae44c14fbf90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2mmq"
Oct 30 19:27:08.601082 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:08.601027    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2mmq"
Oct 30 19:27:08.833608 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:08.833533    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92181cc3-83a9-49ee-b61f-892891560df1" path="/var/lib/kubelet/pods/92181cc3-83a9-49ee-b61f-892891560df1/volumes"
Oct 30 19:27:08.965091 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:08.965025    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2mmq"]
Oct 30 19:27:08.967269 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:27:08.967239    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3dd5295_ae7b_40e0_ae62_ae44c14fbf90.slice/crio-4e1fab05b2424ad8190a9efef35de237a8f86bfe635faf3eea6ebcc28a73ce45 WatchSource:0}: Error finding container 4e1fab05b2424ad8190a9efef35de237a8f86bfe635faf3eea6ebcc28a73ce45: Status 404 returned error can't find the container with id 4e1fab05b2424ad8190a9efef35de237a8f86bfe635faf3eea6ebcc28a73ce45
Oct 30 19:27:09.077299 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:09.077268    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2mmq" event={"ID":"d3dd5295-ae7b-40e0-ae62-ae44c14fbf90","Type":"ContainerStarted","Data":"4e1fab05b2424ad8190a9efef35de237a8f86bfe635faf3eea6ebcc28a73ce45"}
Oct 30 19:27:10.086824 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:10.086734    4924 generic.go:356] "Generic (PLEG): container finished" podID="d3dd5295-ae7b-40e0-ae62-ae44c14fbf90" containerID="3f88422b36ca85663c1da8a6062e11e0799615b36a73050cc83edfdf31c16314" exitCode=0
Oct 30 19:27:10.087224 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:10.086817    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2mmq" event={"ID":"d3dd5295-ae7b-40e0-ae62-ae44c14fbf90","Type":"ContainerDied","Data":"3f88422b36ca85663c1da8a6062e11e0799615b36a73050cc83edfdf31c16314"}
Oct 30 19:27:11.234912 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:11.234885    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2mmq"
Oct 30 19:27:11.293685 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:11.293657    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3dd5295-ae7b-40e0-ae62-ae44c14fbf90-kube-api-access-t9p8n\" (UniqueName: \"kubernetes.io/projected/d3dd5295-ae7b-40e0-ae62-ae44c14fbf90-kube-api-access-t9p8n\") pod \"d3dd5295-ae7b-40e0-ae62-ae44c14fbf90\" (UID: \"d3dd5295-ae7b-40e0-ae62-ae44c14fbf90\") "
Oct 30 19:27:11.293887 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:11.293706    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3dd5295-ae7b-40e0-ae62-ae44c14fbf90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3dd5295-ae7b-40e0-ae62-ae44c14fbf90-maas-api-xks-ca\") pod \"d3dd5295-ae7b-40e0-ae62-ae44c14fbf90\" (UID: \"d3dd5295-ae7b-40e0-ae62-ae44c14fbf90\") "
Oct 30 19:27:11.293887 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:11.293734    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3dd5295-ae7b-40e0-ae62-ae44c14fbf90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3dd5295-ae7b-40e0-ae62-ae44c14fbf90-maas-api-openshift-ca\") pod \"d3dd5295-ae7b-40e0-ae62-ae44c14fbf90\" (UID: \"d3dd5295-ae7b-40e0-ae62-ae44c14fbf90\") "
Oct 30 19:27:11.294187 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:11.294162    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3dd5295-ae7b-40e0-ae62-ae44c14fbf90-maas-api-openshift-ca" pod "d3dd5295-ae7b-40e0-ae62-ae44c14fbf90" (UID: "d3dd5295-ae7b-40e0-ae62-ae44c14fbf90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:11.295777 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:11.295758    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3dd5295-ae7b-40e0-ae62-ae44c14fbf90-maas-api-xks-ca" pod "d3dd5295-ae7b-40e0-ae62-ae44c14fbf90" (UID: "d3dd5295-ae7b-40e0-ae62-ae44c14fbf90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:11.295833 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:11.295811    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3dd5295-ae7b-40e0-ae62-ae44c14fbf90-kube-api-access-t9p8n" pod "d3dd5295-ae7b-40e0-ae62-ae44c14fbf90" (UID: "d3dd5295-ae7b-40e0-ae62-ae44c14fbf90"). InnerVolumeSpecName "kube-api-access-t9p8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:11.394671 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:11.394594    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3dd5295-ae7b-40e0-ae62-ae44c14fbf90-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:11.394671 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:11.394626    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3dd5295-ae7b-40e0-ae62-ae44c14fbf90-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:11.394671 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:11.394637    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9p8n\" (UniqueName: \"kubernetes.io/projected/d3dd5295-ae7b-40e0-ae62-ae44c14fbf90-kube-api-access-t9p8n\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:12.046926 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:12.046890    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-825zc"]
Oct 30 19:27:12.093776 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:12.093741    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-825zc"]
Oct 30 19:27:12.099660 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:12.099634    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2mmq"
Oct 30 19:27:12.099881 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:12.099669    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2mmq" event={"ID":"d3dd5295-ae7b-40e0-ae62-ae44c14fbf90","Type":"ContainerDied","Data":"4e1fab05b2424ad8190a9efef35de237a8f86bfe635faf3eea6ebcc28a73ce45"}
Oct 30 19:27:12.099881 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:12.099739    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e1fab05b2424ad8190a9efef35de237a8f86bfe635faf3eea6ebcc28a73ce45"
Oct 30 19:27:12.833202 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:12.833159    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a6a2041-f7ce-4f30-a120-cd3b77da55a8" path="/var/lib/kubelet/pods/9a6a2041-f7ce-4f30-a120-cd3b77da55a8/volumes"
Oct 30 19:27:15.710248 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:27:15.710212    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:27:41.804391 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:41.804350    4924 scope.go:122] "RemoveContainer" containerID="9427cc0d45d7a348c0c9a607032b2fa3aeb3cb9b1e7366f183e2d657f679f79e"
Oct 30 19:27:41.820033 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:41.820009    4924 scope.go:122] "RemoveContainer" containerID="5d1425fd456b1aa0492e0a1975166984763c4bfd511f790bc8a228a6f2177599"
Oct 30 19:27:42.297543 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:42.297498    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2mmq"]
Oct 30 19:27:42.344060 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:42.344032    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2mmq"]
Oct 30 19:27:42.419685 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:42.419655    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkbn4"]
Oct 30 19:27:42.420087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:42.420049    4924 cpu_manager.go:414] "containerMap: removing container" podUID="d3dd5295-ae7b-40e0-ae62-ae44c14fbf90" containerName="revoke-keys"
Oct 30 19:27:42.420087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:42.420077    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3dd5295-ae7b-40e0-ae62-ae44c14fbf90" containerName="revoke-keys"
Oct 30 19:27:42.420186 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:42.420165    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3dd5295-ae7b-40e0-ae62-ae44c14fbf90" containerName="revoke-keys"
Oct 30 19:27:42.424923 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:42.424904    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkbn4"
Oct 30 19:27:42.475689 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:42.475652    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:27:42.475892 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:42.475764    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:27:42.476026 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:42.475780    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:27:42.476166 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:42.475787    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:27:42.478557 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:42.478536    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkbn4"]
Oct 30 19:27:42.490819 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:42.490793    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4b21c56-96fd-4356-b7b7-1c314e73ca63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lkbn4\" (UID: \"a4b21c56-96fd-4356-b7b7-1c314e73ca63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkbn4"
Oct 30 19:27:42.490934 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:42.490844    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4b21c56-96fd-4356-b7b7-1c314e73ca63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lkbn4\" (UID: \"a4b21c56-96fd-4356-b7b7-1c314e73ca63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkbn4"
Oct 30 19:27:42.491006 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:42.490952    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cscpj\" (UniqueName: \"kubernetes.io/projected/a4b21c56-96fd-4356-b7b7-1c314e73ca63-kube-api-access-cscpj\") pod \"maas-api-revoke-sub-3d4156f1-lkbn4\" (UID: \"a4b21c56-96fd-4356-b7b7-1c314e73ca63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkbn4"
Oct 30 19:27:42.592052 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:42.591982    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cscpj\" (UniqueName: \"kubernetes.io/projected/a4b21c56-96fd-4356-b7b7-1c314e73ca63-kube-api-access-cscpj\") pod \"maas-api-revoke-sub-3d4156f1-lkbn4\" (UID: \"a4b21c56-96fd-4356-b7b7-1c314e73ca63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkbn4"
Oct 30 19:27:42.592210 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:42.592088    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4b21c56-96fd-4356-b7b7-1c314e73ca63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lkbn4\" (UID: \"a4b21c56-96fd-4356-b7b7-1c314e73ca63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkbn4"
Oct 30 19:27:42.592210 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:42.592156    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4b21c56-96fd-4356-b7b7-1c314e73ca63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lkbn4\" (UID: \"a4b21c56-96fd-4356-b7b7-1c314e73ca63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkbn4"
Oct 30 19:27:42.592811 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:42.592794    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4b21c56-96fd-4356-b7b7-1c314e73ca63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lkbn4\" (UID: \"a4b21c56-96fd-4356-b7b7-1c314e73ca63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkbn4"
Oct 30 19:27:42.594399 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:42.594381    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4b21c56-96fd-4356-b7b7-1c314e73ca63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lkbn4\" (UID: \"a4b21c56-96fd-4356-b7b7-1c314e73ca63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkbn4"
Oct 30 19:27:42.647518 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:42.647491    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cscpj\" (UniqueName: \"kubernetes.io/projected/a4b21c56-96fd-4356-b7b7-1c314e73ca63-kube-api-access-cscpj\") pod \"maas-api-revoke-sub-3d4156f1-lkbn4\" (UID: \"a4b21c56-96fd-4356-b7b7-1c314e73ca63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkbn4"
Oct 30 19:27:42.737549 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:42.737524    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkbn4"
Oct 30 19:27:42.835218 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:42.835184    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3dd5295-ae7b-40e0-ae62-ae44c14fbf90" path="/var/lib/kubelet/pods/d3dd5295-ae7b-40e0-ae62-ae44c14fbf90/volumes"
Oct 30 19:27:43.103852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:43.103824    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkbn4"]
Oct 30 19:27:43.106191 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:27:43.106158    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b21c56_96fd_4356_b7b7_1c314e73ca63.slice/crio-45f3dc0c1501e9c3740c2c50132a692479325d3685de8f686ee05b9feee13c60 WatchSource:0}: Error finding container 45f3dc0c1501e9c3740c2c50132a692479325d3685de8f686ee05b9feee13c60: Status 404 returned error can't find the container with id 45f3dc0c1501e9c3740c2c50132a692479325d3685de8f686ee05b9feee13c60
Oct 30 19:27:43.264634 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:43.264597    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkbn4" event={"ID":"a4b21c56-96fd-4356-b7b7-1c314e73ca63","Type":"ContainerStarted","Data":"45f3dc0c1501e9c3740c2c50132a692479325d3685de8f686ee05b9feee13c60"}
Oct 30 19:27:44.271366 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:44.271288    4924 generic.go:356] "Generic (PLEG): container finished" podID="a4b21c56-96fd-4356-b7b7-1c314e73ca63" containerID="699c5e5b756aaf1ab81cb39078f8ef856229b0e3b7ffed12303d290d33e1002a" exitCode=0
Oct 30 19:27:44.271738 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:44.271377    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkbn4" event={"ID":"a4b21c56-96fd-4356-b7b7-1c314e73ca63","Type":"ContainerDied","Data":"699c5e5b756aaf1ab81cb39078f8ef856229b0e3b7ffed12303d290d33e1002a"}
Oct 30 19:27:45.412611 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:45.412585    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkbn4"
Oct 30 19:27:45.516545 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:45.516503    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:45.525647 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:45.525621    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4b21c56-96fd-4356-b7b7-1c314e73ca63-kube-api-access-cscpj\" (UniqueName: \"kubernetes.io/projected/a4b21c56-96fd-4356-b7b7-1c314e73ca63-kube-api-access-cscpj\") pod \"a4b21c56-96fd-4356-b7b7-1c314e73ca63\" (UID: \"a4b21c56-96fd-4356-b7b7-1c314e73ca63\") "
Oct 30 19:27:45.525796 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:45.525731    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4b21c56-96fd-4356-b7b7-1c314e73ca63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4b21c56-96fd-4356-b7b7-1c314e73ca63-maas-api-xks-ca\") pod \"a4b21c56-96fd-4356-b7b7-1c314e73ca63\" (UID: \"a4b21c56-96fd-4356-b7b7-1c314e73ca63\") "
Oct 30 19:27:45.525796 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:45.525765    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4b21c56-96fd-4356-b7b7-1c314e73ca63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4b21c56-96fd-4356-b7b7-1c314e73ca63-maas-api-openshift-ca\") pod \"a4b21c56-96fd-4356-b7b7-1c314e73ca63\" (UID: \"a4b21c56-96fd-4356-b7b7-1c314e73ca63\") "
Oct 30 19:27:45.526191 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:45.526167    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4b21c56-96fd-4356-b7b7-1c314e73ca63-maas-api-openshift-ca" pod "a4b21c56-96fd-4356-b7b7-1c314e73ca63" (UID: "a4b21c56-96fd-4356-b7b7-1c314e73ca63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:45.529927 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:45.529868    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b21c56-96fd-4356-b7b7-1c314e73ca63-kube-api-access-cscpj" pod "a4b21c56-96fd-4356-b7b7-1c314e73ca63" (UID: "a4b21c56-96fd-4356-b7b7-1c314e73ca63"). InnerVolumeSpecName "kube-api-access-cscpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:45.530047 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:45.529930    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4b21c56-96fd-4356-b7b7-1c314e73ca63-maas-api-xks-ca" pod "a4b21c56-96fd-4356-b7b7-1c314e73ca63" (UID: "a4b21c56-96fd-4356-b7b7-1c314e73ca63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:45.627128 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:45.627085    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cscpj\" (UniqueName: \"kubernetes.io/projected/a4b21c56-96fd-4356-b7b7-1c314e73ca63-kube-api-access-cscpj\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:45.627128 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:45.627125    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4b21c56-96fd-4356-b7b7-1c314e73ca63-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:45.627128 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:45.627136    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4b21c56-96fd-4356-b7b7-1c314e73ca63-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:45.710766 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:27:45.710727    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:27:46.283024 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:46.282981    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkbn4" event={"ID":"a4b21c56-96fd-4356-b7b7-1c314e73ca63","Type":"ContainerDied","Data":"45f3dc0c1501e9c3740c2c50132a692479325d3685de8f686ee05b9feee13c60"}
Oct 30 19:27:46.283220 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:46.283045    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45f3dc0c1501e9c3740c2c50132a692479325d3685de8f686ee05b9feee13c60"
Oct 30 19:27:46.283220 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:27:46.282994    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkbn4"
Oct 30 19:28:09.048387 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:09.048352    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnw6b"]
Oct 30 19:28:09.095094 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:09.095048    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pnw6b"]
Oct 30 19:28:10.833205 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:10.833161    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c84fb511-1497-447c-9691-5456345073dc" path="/var/lib/kubelet/pods/c84fb511-1497-447c-9691-5456345073dc/volumes"
Oct 30 19:28:11.278938 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:11.278909    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:12.075768 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:12.075732    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:12.435757 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:12.435724    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:15.710470 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:28:15.710433    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:28:16.734542 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:16.734505    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:17.038792 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:17.038701    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:19.873302 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:19.873226    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:20.186514 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:20.186424    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:20.445953 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:20.445877    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg"]
Oct 30 19:28:20.446320 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:20.446305    4924 cpu_manager.go:414] "containerMap: removing container" podUID="a4b21c56-96fd-4356-b7b7-1c314e73ca63" containerName="revoke-keys"
Oct 30 19:28:20.446381 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:20.446321    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4b21c56-96fd-4356-b7b7-1c314e73ca63" containerName="revoke-keys"
Oct 30 19:28:20.446438 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:20.446391    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4b21c56-96fd-4356-b7b7-1c314e73ca63" containerName="revoke-keys"
Oct 30 19:28:20.450744 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:20.450723    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg"
Oct 30 19:28:20.501516 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:20.501485    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-7d30a0\""
Oct 30 19:28:20.501733 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:20.501537    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-721d9a0c-dockercfg-b9qll\""
Oct 30 19:28:20.504184 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:20.504160    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg"]
Oct 30 19:28:20.566088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:20.566030    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5cmq\" (UniqueName: \"kubernetes.io/projected/d3f85573-6434-4b07-b67e-396744b7d8bb-kube-api-access-g5cmq\") pod \"payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg\" (UID: \"d3f85573-6434-4b07-b67e-396744b7d8bb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg"
Oct 30 19:28:20.566262 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:20.566207    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f85573-6434-4b07-b67e-396744b7d8bb-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg\" (UID: \"d3f85573-6434-4b07-b67e-396744b7d8bb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg"
Oct 30 19:28:20.667444 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:20.667406    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5cmq\" (UniqueName: \"kubernetes.io/projected/d3f85573-6434-4b07-b67e-396744b7d8bb-kube-api-access-g5cmq\") pod \"payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg\" (UID: \"d3f85573-6434-4b07-b67e-396744b7d8bb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg"
Oct 30 19:28:20.667612 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:20.667542    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f85573-6434-4b07-b67e-396744b7d8bb-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg\" (UID: \"d3f85573-6434-4b07-b67e-396744b7d8bb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg"
Oct 30 19:28:20.668367 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:20.668347    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f85573-6434-4b07-b67e-396744b7d8bb-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg\" (UID: \"d3f85573-6434-4b07-b67e-396744b7d8bb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg"
Oct 30 19:28:20.722293 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:20.722232    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5cmq\" (UniqueName: \"kubernetes.io/projected/d3f85573-6434-4b07-b67e-396744b7d8bb-kube-api-access-g5cmq\") pod \"payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg\" (UID: \"d3f85573-6434-4b07-b67e-396744b7d8bb\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg"
Oct 30 19:28:20.762956 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:20.762925    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg"
Oct 30 19:28:21.130359 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:21.130326    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg"]
Oct 30 19:28:21.131854 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:28:21.131799    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f85573_6434_4b07_b67e_396744b7d8bb.slice/crio-2e97188d0116f0c16980c6f9f6b55934fa862dd7075a2da01578a5aea84147d9 WatchSource:0}: Error finding container 2e97188d0116f0c16980c6f9f6b55934fa862dd7075a2da01578a5aea84147d9: Status 404 returned error can't find the container with id 2e97188d0116f0c16980c6f9f6b55934fa862dd7075a2da01578a5aea84147d9
Oct 30 19:28:21.478246 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:21.478161    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg" event={"ID":"d3f85573-6434-4b07-b67e-396744b7d8bb","Type":"ContainerStarted","Data":"d6b2ed5feed56bbcbfa1b9fdbbd44c134fa460226ee1edb53d91dbcdd082ca39"}
Oct 30 19:28:21.478246 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:21.478214    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg" event={"ID":"d3f85573-6434-4b07-b67e-396744b7d8bb","Type":"ContainerStarted","Data":"2e97188d0116f0c16980c6f9f6b55934fa862dd7075a2da01578a5aea84147d9"}
Oct 30 19:28:21.478447 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:21.478280    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg"
Oct 30 19:28:23.232618 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:23.232586    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:23.566533 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:23.566454    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:28.248718 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:28.248682    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:28.932956 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:28.932913    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:32.487141 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:32.487111    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg"
Oct 30 19:28:32.595200 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:32.595145    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg" podStartSLOduration=12.595128966 podStartE2EDuration="12.595128966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:21.585961245 +0000 UTC m=+3102.863011701" watchObservedRunningTime="2026-10-30 19:28:32.595128966 +0000 UTC m=+3113.872179412"
Oct 30 19:28:33.246654 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:33.246614    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:38.163846 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:38.163809    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:41.048515 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:41.048475    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tqs5k"]
Oct 30 19:28:41.095329 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:41.095292    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tqs5k"]
Oct 30 19:28:41.613871 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:41.613827    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:41.847336 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:41.847304    4924 scope.go:122] "RemoveContainer" containerID="cdf2e281254a892508470c5332bafcfc0a8e3f4151147e9a90731fb7e929ade9"
Oct 30 19:28:42.245323 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:42.245291    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:42.657816 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:42.657782    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:42.833486 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:42.833449    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1825e05b-4412-4cfc-bfd0-a0feb5e472fd" path="/var/lib/kubelet/pods/1825e05b-4412-4cfc-bfd0-a0feb5e472fd/volumes"
Oct 30 19:28:43.285043 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:43.285009    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:45.710719 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:28:45.710684    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:28:45.787170 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:45.787138    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:45.988963 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:45.988891    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"]
Oct 30 19:28:45.993730 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:45.993711    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"
Oct 30 19:28:46.075704 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:46.075673    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"]
Oct 30 19:28:46.092546 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:46.092508    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-7d30a0\"/\"kube-root-ca.crt\""
Oct 30 19:28:46.092546 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:46.092539    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-7d30a0\"/\"default-dockercfg-ngdpc\""
Oct 30 19:28:46.092731 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:46.092552    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-7d30a0\"/\"simulator-single-main-7d30a0-kserve-self-signed-certs\""
Oct 30 19:28:46.092731 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:46.092558    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-7d30a0\"/\"openshift-service-ca.crt\""
Oct 30 19:28:46.116690 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:46.116662    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-dshm\") pod \"simulator-single-main-7d30a0-kserve-5945d949bc-t8824\" (UID: \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\") " pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"
Oct 30 19:28:46.116810 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:46.116696    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-model-cache\") pod \"simulator-single-main-7d30a0-kserve-5945d949bc-t8824\" (UID: \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\") " pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"
Oct 30 19:28:46.116810 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:46.116719    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-tmp-dir\") pod \"simulator-single-main-7d30a0-kserve-5945d949bc-t8824\" (UID: \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\") " pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"
Oct 30 19:28:46.116895 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:46.116854    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-tls-certs\") pod \"simulator-single-main-7d30a0-kserve-5945d949bc-t8824\" (UID: \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\") " pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"
Oct 30 19:28:46.116934 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:46.116900    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-879tw\" (UniqueName: \"kubernetes.io/projected/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-kube-api-access-879tw\") pod \"simulator-single-main-7d30a0-kserve-5945d949bc-t8824\" (UID: \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\") " pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"
Oct 30 19:28:46.117000 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:46.116986    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-home\") pod \"simulator-single-main-7d30a0-kserve-5945d949bc-t8824\" (UID: \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\") " pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"
Oct 30 19:28:46.136226 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:46.136202    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:46.218346 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:46.218315    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-tls-certs\") pod \"simulator-single-main-7d30a0-kserve-5945d949bc-t8824\" (UID: \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\") " pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"
Oct 30 19:28:46.218528 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:46.218363    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-879tw\" (UniqueName: \"kubernetes.io/projected/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-kube-api-access-879tw\") pod \"simulator-single-main-7d30a0-kserve-5945d949bc-t8824\" (UID: \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\") " pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"
Oct 30 19:28:46.218528 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:46.218509    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-home\") pod \"simulator-single-main-7d30a0-kserve-5945d949bc-t8824\" (UID: \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\") " pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"
Oct 30 19:28:46.218620 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:46.218585    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-dshm\") pod \"simulator-single-main-7d30a0-kserve-5945d949bc-t8824\" (UID: \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\") " pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"
Oct 30 19:28:46.218679 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:46.218618    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-model-cache\") pod \"simulator-single-main-7d30a0-kserve-5945d949bc-t8824\" (UID: \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\") " pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"
Oct 30 19:28:46.218740 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:46.218682    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-tmp-dir\") pod \"simulator-single-main-7d30a0-kserve-5945d949bc-t8824\" (UID: \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\") " pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"
Oct 30 19:28:46.218967 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:46.218944    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-home\") pod \"simulator-single-main-7d30a0-kserve-5945d949bc-t8824\" (UID: \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\") " pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"
Oct 30 19:28:46.219080 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:46.218975    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-model-cache\") pod \"simulator-single-main-7d30a0-kserve-5945d949bc-t8824\" (UID: \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\") " pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"
Oct 30 19:28:46.219080 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:46.219032    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-tmp-dir\") pod \"simulator-single-main-7d30a0-kserve-5945d949bc-t8824\" (UID: \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\") " pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"
Oct 30 19:28:46.220722 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:46.220704    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-dshm\") pod \"simulator-single-main-7d30a0-kserve-5945d949bc-t8824\" (UID: \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\") " pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"
Oct 30 19:28:46.220809 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:46.220793    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-tls-certs\") pod \"simulator-single-main-7d30a0-kserve-5945d949bc-t8824\" (UID: \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\") " pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"
Oct 30 19:28:46.272759 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:46.272693    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-879tw\" (UniqueName: \"kubernetes.io/projected/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-kube-api-access-879tw\") pod \"simulator-single-main-7d30a0-kserve-5945d949bc-t8824\" (UID: \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\") " pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"
Oct 30 19:28:46.305124 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:46.305095    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"
Oct 30 19:28:46.669465 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:46.669442    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"]
Oct 30 19:28:46.671264 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:28:46.671226    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc970b97_eec9_414e_8ac6_c36ea3e2f6ff.slice/crio-ab616e5b23e464befeb8492841eef4ea27cdda81ccde9a6753039f900f01b682 WatchSource:0}: Error finding container ab616e5b23e464befeb8492841eef4ea27cdda81ccde9a6753039f900f01b682: Status 404 returned error can't find the container with id ab616e5b23e464befeb8492841eef4ea27cdda81ccde9a6753039f900f01b682
Oct 30 19:28:47.615896 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:47.615859    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824" event={"ID":"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff","Type":"ContainerStarted","Data":"3f2b3aea7bd8cb2841e45336b048ca2d4cefeb13c14a89fc4bc833735f1c5e3a"}
Oct 30 19:28:47.615896 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:47.615902    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824" event={"ID":"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff","Type":"ContainerStarted","Data":"ab616e5b23e464befeb8492841eef4ea27cdda81ccde9a6753039f900f01b682"}
Oct 30 19:28:47.724835 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:47.724777    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824" podStartSLOduration=2.7247596769999998 podStartE2EDuration="2.724759677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:47.724139951 +0000 UTC m=+3129.001190396" watchObservedRunningTime="2026-10-30 19:28:47.724759677 +0000 UTC m=+3129.001810120"
Oct 30 19:28:48.170247 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:48.170215    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:48.305934 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:48.305897    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"
Oct 30 19:28:52.982340 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:52.982294    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:53.047952 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:53.047916    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hgw5n"]
Oct 30 19:28:53.093590 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:53.093559    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hgw5n"]
Oct 30 19:28:54.833174 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:54.833135    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95d071ca-41b3-46b5-9175-881954a9fdef" path="/var/lib/kubelet/pods/95d071ca-41b3-46b5-9175-881954a9fdef/volumes"
Oct 30 19:28:56.306020 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:56.305985    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"
Oct 30 19:28:56.318712 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:56.318691    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"
Oct 30 19:28:56.674433 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:56.674395    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"
Oct 30 19:28:57.826911 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:28:57.826873    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:02.647045 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:02.647007    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:07.596458 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:07.592179    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:12.475172 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:12.475137    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:15.710964 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:29:15.710930    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:29:17.447438 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:17.447404    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:21.604656 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:21.604615    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7de1cfd4-7vghp"]
Oct 30 19:29:21.608542 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:21.608520    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de1cfd4-7vghp"
Oct 30 19:29:21.708235 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:21.708199    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:29:21.708412 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:21.708394    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:29:21.708459 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:21.708422    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:29:21.708495 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:21.708481    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:29:21.710632 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:21.710610    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7de1cfd4-7vghp"]
Oct 30 19:29:21.749070 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:21.749043    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d563938-c50f-47b4-b74f-1ae0648085fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7de1cfd4-7vghp\" (UID: \"2d563938-c50f-47b4-b74f-1ae0648085fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de1cfd4-7vghp"
Oct 30 19:29:21.749227 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:21.749122    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p57np\" (UniqueName: \"kubernetes.io/projected/2d563938-c50f-47b4-b74f-1ae0648085fd-kube-api-access-p57np\") pod \"maas-api-revoke-sub-7de1cfd4-7vghp\" (UID: \"2d563938-c50f-47b4-b74f-1ae0648085fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de1cfd4-7vghp"
Oct 30 19:29:21.749227 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:21.749211    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d563938-c50f-47b4-b74f-1ae0648085fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7de1cfd4-7vghp\" (UID: \"2d563938-c50f-47b4-b74f-1ae0648085fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de1cfd4-7vghp"
Oct 30 19:29:21.849820 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:21.849788    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d563938-c50f-47b4-b74f-1ae0648085fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7de1cfd4-7vghp\" (UID: \"2d563938-c50f-47b4-b74f-1ae0648085fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de1cfd4-7vghp"
Oct 30 19:29:21.850005 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:21.849849    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p57np\" (UniqueName: \"kubernetes.io/projected/2d563938-c50f-47b4-b74f-1ae0648085fd-kube-api-access-p57np\") pod \"maas-api-revoke-sub-7de1cfd4-7vghp\" (UID: \"2d563938-c50f-47b4-b74f-1ae0648085fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de1cfd4-7vghp"
Oct 30 19:29:21.850005 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:21.849878    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d563938-c50f-47b4-b74f-1ae0648085fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7de1cfd4-7vghp\" (UID: \"2d563938-c50f-47b4-b74f-1ae0648085fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de1cfd4-7vghp"
Oct 30 19:29:21.850517 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:21.850489    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d563938-c50f-47b4-b74f-1ae0648085fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7de1cfd4-7vghp\" (UID: \"2d563938-c50f-47b4-b74f-1ae0648085fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de1cfd4-7vghp"
Oct 30 19:29:21.852219 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:21.852199    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d563938-c50f-47b4-b74f-1ae0648085fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7de1cfd4-7vghp\" (UID: \"2d563938-c50f-47b4-b74f-1ae0648085fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de1cfd4-7vghp"
Oct 30 19:29:21.905422 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:21.905353    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p57np\" (UniqueName: \"kubernetes.io/projected/2d563938-c50f-47b4-b74f-1ae0648085fd-kube-api-access-p57np\") pod \"maas-api-revoke-sub-7de1cfd4-7vghp\" (UID: \"2d563938-c50f-47b4-b74f-1ae0648085fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de1cfd4-7vghp"
Oct 30 19:29:21.922141 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:21.922114    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de1cfd4-7vghp"
Oct 30 19:29:22.286717 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:22.286683    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7de1cfd4-7vghp"]
Oct 30 19:29:22.288352 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:29:22.288320    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d563938_c50f_47b4_b74f_1ae0648085fd.slice/crio-c4d3772854d72e81438bad16e033041291b8439276c71399e72fd05bb5b464f1 WatchSource:0}: Error finding container c4d3772854d72e81438bad16e033041291b8439276c71399e72fd05bb5b464f1: Status 404 returned error can't find the container with id c4d3772854d72e81438bad16e033041291b8439276c71399e72fd05bb5b464f1
Oct 30 19:29:22.371307 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:22.371274    4924 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:22.797663 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:22.797629    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de1cfd4-7vghp" event={"ID":"2d563938-c50f-47b4-b74f-1ae0648085fd","Type":"ContainerStarted","Data":"cca84ec456dba2c3fecb077fcf3a2ff9ab1af7dc75fe7cbf75e449f14fa01355"}
Oct 30 19:29:22.798163 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:22.797689    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de1cfd4-7vghp" event={"ID":"2d563938-c50f-47b4-b74f-1ae0648085fd","Type":"ContainerStarted","Data":"c4d3772854d72e81438bad16e033041291b8439276c71399e72fd05bb5b464f1"}
Oct 30 19:29:22.906964 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:22.906871    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de1cfd4-7vghp" podStartSLOduration=1.90685789 podStartE2EDuration="1.90685789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:22.906180276 +0000 UTC m=+3164.183230722" watchObservedRunningTime="2026-10-30 19:29:22.90685789 +0000 UTC m=+3164.183908337"
Oct 30 19:29:23.804580 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:23.804545    4924 generic.go:356] "Generic (PLEG): container finished" podID="2d563938-c50f-47b4-b74f-1ae0648085fd" containerID="cca84ec456dba2c3fecb077fcf3a2ff9ab1af7dc75fe7cbf75e449f14fa01355" exitCode=0
Oct 30 19:29:23.804976 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:23.804604    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de1cfd4-7vghp" event={"ID":"2d563938-c50f-47b4-b74f-1ae0648085fd","Type":"ContainerDied","Data":"cca84ec456dba2c3fecb077fcf3a2ff9ab1af7dc75fe7cbf75e449f14fa01355"}
Oct 30 19:29:24.957032 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:24.956998    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de1cfd4-7vghp"
Oct 30 19:29:25.078805 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:25.078727    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d563938-c50f-47b4-b74f-1ae0648085fd-kube-api-access-p57np\" (UniqueName: \"kubernetes.io/projected/2d563938-c50f-47b4-b74f-1ae0648085fd-kube-api-access-p57np\") pod \"2d563938-c50f-47b4-b74f-1ae0648085fd\" (UID: \"2d563938-c50f-47b4-b74f-1ae0648085fd\") "
Oct 30 19:29:25.078958 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:25.078821    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d563938-c50f-47b4-b74f-1ae0648085fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d563938-c50f-47b4-b74f-1ae0648085fd-maas-api-xks-ca\") pod \"2d563938-c50f-47b4-b74f-1ae0648085fd\" (UID: \"2d563938-c50f-47b4-b74f-1ae0648085fd\") "
Oct 30 19:29:25.078958 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:25.078934    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d563938-c50f-47b4-b74f-1ae0648085fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d563938-c50f-47b4-b74f-1ae0648085fd-maas-api-openshift-ca\") pod \"2d563938-c50f-47b4-b74f-1ae0648085fd\" (UID: \"2d563938-c50f-47b4-b74f-1ae0648085fd\") "
Oct 30 19:29:25.079370 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:25.079340    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d563938-c50f-47b4-b74f-1ae0648085fd-maas-api-openshift-ca" pod "2d563938-c50f-47b4-b74f-1ae0648085fd" (UID: "2d563938-c50f-47b4-b74f-1ae0648085fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:25.081023 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:25.080989    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d563938-c50f-47b4-b74f-1ae0648085fd-maas-api-xks-ca" pod "2d563938-c50f-47b4-b74f-1ae0648085fd" (UID: "2d563938-c50f-47b4-b74f-1ae0648085fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:25.081180 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:25.080996    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d563938-c50f-47b4-b74f-1ae0648085fd-kube-api-access-p57np" pod "2d563938-c50f-47b4-b74f-1ae0648085fd" (UID: "2d563938-c50f-47b4-b74f-1ae0648085fd"). InnerVolumeSpecName "kube-api-access-p57np". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:25.179988 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:25.179950    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d563938-c50f-47b4-b74f-1ae0648085fd-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:25.179988 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:25.179983    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p57np\" (UniqueName: \"kubernetes.io/projected/2d563938-c50f-47b4-b74f-1ae0648085fd-kube-api-access-p57np\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:25.179988 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:25.179994    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d563938-c50f-47b4-b74f-1ae0648085fd-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:25.816285 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:25.816246    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de1cfd4-7vghp" event={"ID":"2d563938-c50f-47b4-b74f-1ae0648085fd","Type":"ContainerDied","Data":"c4d3772854d72e81438bad16e033041291b8439276c71399e72fd05bb5b464f1"}
Oct 30 19:29:25.816285 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:25.816276    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7de1cfd4-7vghp"
Oct 30 19:29:25.816506 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:25.816313    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4d3772854d72e81438bad16e033041291b8439276c71399e72fd05bb5b464f1"
Oct 30 19:29:29.099267 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:29.099229    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kwljn"]
Oct 30 19:29:29.144238 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:29.144206    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kwljn"]
Oct 30 19:29:30.837669 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:30.837635    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f4b913f-25d9-492a-879c-022484c1340f" path="/var/lib/kubelet/pods/2f4b913f-25d9-492a-879c-022484c1340f/volumes"
Oct 30 19:29:33.739952 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:33.739914    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t"]
Oct 30 19:29:33.740399 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:33.740335    4924 cpu_manager.go:414] "containerMap: removing container" podUID="2d563938-c50f-47b4-b74f-1ae0648085fd" containerName="revoke-keys"
Oct 30 19:29:33.740399 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:33.740348    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d563938-c50f-47b4-b74f-1ae0648085fd" containerName="revoke-keys"
Oct 30 19:29:33.740487 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:33.740421    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d563938-c50f-47b4-b74f-1ae0648085fd" containerName="revoke-keys"
Oct 30 19:29:33.743873 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:33.743851    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t"
Oct 30 19:29:33.799414 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:33.799384    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t"]
Oct 30 19:29:33.843764 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:33.843739    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:29:33.843922 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:33.843764    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:29:33.843922 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:33.843795    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:29:33.843922 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:33.843890    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:29:33.864994 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:33.864965    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63f3c6ad-febd-4b14-aaab-d73b7e8f53ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t\" (UID: \"63f3c6ad-febd-4b14-aaab-d73b7e8f53ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t"
Oct 30 19:29:33.864994 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:33.864997    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmqd7\" (UniqueName: \"kubernetes.io/projected/63f3c6ad-febd-4b14-aaab-d73b7e8f53ba-kube-api-access-qmqd7\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t\" (UID: \"63f3c6ad-febd-4b14-aaab-d73b7e8f53ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t"
Oct 30 19:29:33.865182 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:33.865023    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63f3c6ad-febd-4b14-aaab-d73b7e8f53ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t\" (UID: \"63f3c6ad-febd-4b14-aaab-d73b7e8f53ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t"
Oct 30 19:29:33.966254 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:33.966222    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63f3c6ad-febd-4b14-aaab-d73b7e8f53ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t\" (UID: \"63f3c6ad-febd-4b14-aaab-d73b7e8f53ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t"
Oct 30 19:29:33.966436 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:33.966414    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmqd7\" (UniqueName: \"kubernetes.io/projected/63f3c6ad-febd-4b14-aaab-d73b7e8f53ba-kube-api-access-qmqd7\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t\" (UID: \"63f3c6ad-febd-4b14-aaab-d73b7e8f53ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t"
Oct 30 19:29:33.966517 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:33.966484    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63f3c6ad-febd-4b14-aaab-d73b7e8f53ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t\" (UID: \"63f3c6ad-febd-4b14-aaab-d73b7e8f53ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t"
Oct 30 19:29:33.966922 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:33.966897    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63f3c6ad-febd-4b14-aaab-d73b7e8f53ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t\" (UID: \"63f3c6ad-febd-4b14-aaab-d73b7e8f53ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t"
Oct 30 19:29:33.968771 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:33.968751    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63f3c6ad-febd-4b14-aaab-d73b7e8f53ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t\" (UID: \"63f3c6ad-febd-4b14-aaab-d73b7e8f53ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t"
Oct 30 19:29:34.023544 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:34.023479    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmqd7\" (UniqueName: \"kubernetes.io/projected/63f3c6ad-febd-4b14-aaab-d73b7e8f53ba-kube-api-access-qmqd7\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t\" (UID: \"63f3c6ad-febd-4b14-aaab-d73b7e8f53ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t"
Oct 30 19:29:34.056818 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:34.056793    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t"
Oct 30 19:29:34.421451 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:34.421424    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t"]
Oct 30 19:29:34.423333 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:29:34.423303    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f3c6ad_febd_4b14_aaab_d73b7e8f53ba.slice/crio-fac2af9bfb8a356095909fe36e77630246bdf60aeb7f2f49f6f73684409a7fe9 WatchSource:0}: Error finding container fac2af9bfb8a356095909fe36e77630246bdf60aeb7f2f49f6f73684409a7fe9: Status 404 returned error can't find the container with id fac2af9bfb8a356095909fe36e77630246bdf60aeb7f2f49f6f73684409a7fe9
Oct 30 19:29:34.865651 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:34.865618    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t" event={"ID":"63f3c6ad-febd-4b14-aaab-d73b7e8f53ba","Type":"ContainerStarted","Data":"1676385fa104245d0cc13dfff5377397bd9454b664579b32faf3c480bfa4fee0"}
Oct 30 19:29:34.866161 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:34.865683    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t" event={"ID":"63f3c6ad-febd-4b14-aaab-d73b7e8f53ba","Type":"ContainerStarted","Data":"fac2af9bfb8a356095909fe36e77630246bdf60aeb7f2f49f6f73684409a7fe9"}
Oct 30 19:29:34.978187 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:34.977994    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t" podStartSLOduration=1.977979449 podStartE2EDuration="1.977979449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:34.976939805 +0000 UTC m=+3176.253990251" watchObservedRunningTime="2026-10-30 19:29:34.977979449 +0000 UTC m=+3176.255029895"
Oct 30 19:29:35.879918 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:35.879879    4924 generic.go:356] "Generic (PLEG): container finished" podID="63f3c6ad-febd-4b14-aaab-d73b7e8f53ba" containerID="1676385fa104245d0cc13dfff5377397bd9454b664579b32faf3c480bfa4fee0" exitCode=0
Oct 30 19:29:35.880347 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:35.879942    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t" event={"ID":"63f3c6ad-febd-4b14-aaab-d73b7e8f53ba","Type":"ContainerDied","Data":"1676385fa104245d0cc13dfff5377397bd9454b664579b32faf3c480bfa4fee0"}
Oct 30 19:29:37.018710 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:37.018685    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t"
Oct 30 19:29:37.197532 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:37.197436    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63f3c6ad-febd-4b14-aaab-d73b7e8f53ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63f3c6ad-febd-4b14-aaab-d73b7e8f53ba-maas-api-xks-ca\") pod \"63f3c6ad-febd-4b14-aaab-d73b7e8f53ba\" (UID: \"63f3c6ad-febd-4b14-aaab-d73b7e8f53ba\") "
Oct 30 19:29:37.197532 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:37.197494    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63f3c6ad-febd-4b14-aaab-d73b7e8f53ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63f3c6ad-febd-4b14-aaab-d73b7e8f53ba-maas-api-openshift-ca\") pod \"63f3c6ad-febd-4b14-aaab-d73b7e8f53ba\" (UID: \"63f3c6ad-febd-4b14-aaab-d73b7e8f53ba\") "
Oct 30 19:29:37.197774 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:37.197631    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63f3c6ad-febd-4b14-aaab-d73b7e8f53ba-kube-api-access-qmqd7\" (UniqueName: \"kubernetes.io/projected/63f3c6ad-febd-4b14-aaab-d73b7e8f53ba-kube-api-access-qmqd7\") pod \"63f3c6ad-febd-4b14-aaab-d73b7e8f53ba\" (UID: \"63f3c6ad-febd-4b14-aaab-d73b7e8f53ba\") "
Oct 30 19:29:37.197881 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:37.197859    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f3c6ad-febd-4b14-aaab-d73b7e8f53ba-maas-api-openshift-ca" pod "63f3c6ad-febd-4b14-aaab-d73b7e8f53ba" (UID: "63f3c6ad-febd-4b14-aaab-d73b7e8f53ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:29:37.198023 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:37.198000    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63f3c6ad-febd-4b14-aaab-d73b7e8f53ba-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:37.199717 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:37.199694    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63f3c6ad-febd-4b14-aaab-d73b7e8f53ba-maas-api-xks-ca" pod "63f3c6ad-febd-4b14-aaab-d73b7e8f53ba" (UID: "63f3c6ad-febd-4b14-aaab-d73b7e8f53ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:37.199819 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:37.199788    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f3c6ad-febd-4b14-aaab-d73b7e8f53ba-kube-api-access-qmqd7" pod "63f3c6ad-febd-4b14-aaab-d73b7e8f53ba" (UID: "63f3c6ad-febd-4b14-aaab-d73b7e8f53ba"). InnerVolumeSpecName "kube-api-access-qmqd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:37.299053 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:37.299012    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmqd7\" (UniqueName: \"kubernetes.io/projected/63f3c6ad-febd-4b14-aaab-d73b7e8f53ba-kube-api-access-qmqd7\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:37.299053 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:37.299048    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63f3c6ad-febd-4b14-aaab-d73b7e8f53ba-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:37.890646 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:37.890616    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t"
Oct 30 19:29:37.890858 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:37.890648    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t" event={"ID":"63f3c6ad-febd-4b14-aaab-d73b7e8f53ba","Type":"ContainerDied","Data":"fac2af9bfb8a356095909fe36e77630246bdf60aeb7f2f49f6f73684409a7fe9"}
Oct 30 19:29:37.890858 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:37.890698    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fac2af9bfb8a356095909fe36e77630246bdf60aeb7f2f49f6f73684409a7fe9"
Oct 30 19:29:38.411778 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:38.411742    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"]
Oct 30 19:29:40.047657 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:40.047625    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xqtcb"]
Oct 30 19:29:40.068535 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:40.068509    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-xqtcb"]
Oct 30 19:29:40.833441 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:40.833411    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="822881c0-b131-4f45-82a0-22e1880f482d" path="/var/lib/kubelet/pods/822881c0-b131-4f45-82a0-22e1880f482d/volumes"
Oct 30 19:29:41.869119 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:41.869083    4924 scope.go:122] "RemoveContainer" containerID="f2f6fd31b656c51caa7f1fa620480a518c34f6c487fe5371eb95c0f54dd65ddd"
Oct 30 19:29:41.877998 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:41.877969    4924 scope.go:122] "RemoveContainer" containerID="b2fa5bde1798c5134e14921e6b9d007c724a1e4d3c1a86aabadad2ea02aa8d11"
Oct 30 19:29:45.117304 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:45.117267    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg"]
Oct 30 19:29:45.710663 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:29:45.710627    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:29:50.118293 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:50.118191    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg" podUID="d3f85573-6434-4b07-b67e-396744b7d8bb" containerName="payload-processing" containerID="cri-o://d6b2ed5feed56bbcbfa1b9fdbbd44c134fa460226ee1edb53d91dbcdd082ca39" gracePeriod=25
Oct 30 19:29:52.484934 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:52.484894    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg" podUID="d3f85573-6434-4b07-b67e-396744b7d8bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:29:53.432021 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.431976    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824" podUID="cc970b97-eec9-414e-8ac6-c36ea3e2f6ff" containerName="main" containerID="cri-o://3f2b3aea7bd8cb2841e45336b048ca2d4cefeb13c14a89fc4bc833735f1c5e3a" gracePeriod=45
Oct 30 19:29:53.627485 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.627449    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t"]
Oct 30 19:29:53.673906 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.673876    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7d30a0-nm44t"]
Oct 30 19:29:53.687992 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.687944    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"
Oct 30 19:29:53.752520 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.752477    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-dshm\") pod \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\" (UID: \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\") "
Oct 30 19:29:53.752707 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.752542    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-kube-api-access-879tw\" (UniqueName: \"kubernetes.io/projected/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-kube-api-access-879tw\") pod \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\" (UID: \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\") "
Oct 30 19:29:53.752707 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.752561    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-home\" (UniqueName: \"kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-home\") pod \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\" (UID: \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\") "
Oct 30 19:29:53.752707 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.752613    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-tls-certs\") pod \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\" (UID: \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\") "
Oct 30 19:29:53.752707 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.752631    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-tmp-dir\") pod \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\" (UID: \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\") "
Oct 30 19:29:53.752707 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.752658    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-model-cache\") pod \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\" (UID: \"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff\") "
Oct 30 19:29:53.753019 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.752848    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-home" pod "cc970b97-eec9-414e-8ac6-c36ea3e2f6ff" (UID: "cc970b97-eec9-414e-8ac6-c36ea3e2f6ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:53.753019 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.752914    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-tmp-dir" pod "cc970b97-eec9-414e-8ac6-c36ea3e2f6ff" (UID: "cc970b97-eec9-414e-8ac6-c36ea3e2f6ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:53.753158 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.753040    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-model-cache" pod "cc970b97-eec9-414e-8ac6-c36ea3e2f6ff" (UID: "cc970b97-eec9-414e-8ac6-c36ea3e2f6ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:53.753158 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.753111    4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-home\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:53.753158 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.753125    4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-tmp-dir\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:53.754842 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.754817    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-dshm" pod "cc970b97-eec9-414e-8ac6-c36ea3e2f6ff" (UID: "cc970b97-eec9-414e-8ac6-c36ea3e2f6ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:29:53.755152 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.755126    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-tls-certs" pod "cc970b97-eec9-414e-8ac6-c36ea3e2f6ff" (UID: "cc970b97-eec9-414e-8ac6-c36ea3e2f6ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:29:53.755258 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.755154    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-kube-api-access-879tw" pod "cc970b97-eec9-414e-8ac6-c36ea3e2f6ff" (UID: "cc970b97-eec9-414e-8ac6-c36ea3e2f6ff"). InnerVolumeSpecName "kube-api-access-879tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:29:53.854382 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.854287    4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-dshm\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:53.854382 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.854320    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-879tw\" (UniqueName: \"kubernetes.io/projected/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-kube-api-access-879tw\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:53.854382 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.854336    4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-tls-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:53.854382 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.854351    4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff-model-cache\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:29:53.973341 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.973253    4924 generic.go:356] "Generic (PLEG): container finished" podID="cc970b97-eec9-414e-8ac6-c36ea3e2f6ff" containerID="3f2b3aea7bd8cb2841e45336b048ca2d4cefeb13c14a89fc4bc833735f1c5e3a" exitCode=0
Oct 30 19:29:53.973341 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.973312    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"
Oct 30 19:29:53.973557 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.973333    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824" event={"ID":"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff","Type":"ContainerDied","Data":"3f2b3aea7bd8cb2841e45336b048ca2d4cefeb13c14a89fc4bc833735f1c5e3a"}
Oct 30 19:29:53.973557 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.973379    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824" event={"ID":"cc970b97-eec9-414e-8ac6-c36ea3e2f6ff","Type":"ContainerDied","Data":"ab616e5b23e464befeb8492841eef4ea27cdda81ccde9a6753039f900f01b682"}
Oct 30 19:29:53.973557 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.973395    4924 scope.go:122] "RemoveContainer" containerID="3f2b3aea7bd8cb2841e45336b048ca2d4cefeb13c14a89fc4bc833735f1c5e3a"
Oct 30 19:29:53.983513 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.983495    4924 scope.go:122] "RemoveContainer" containerID="3f2b3aea7bd8cb2841e45336b048ca2d4cefeb13c14a89fc4bc833735f1c5e3a"
Oct 30 19:29:53.983778 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:29:53.983759    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f2b3aea7bd8cb2841e45336b048ca2d4cefeb13c14a89fc4bc833735f1c5e3a\": container with ID starting with 3f2b3aea7bd8cb2841e45336b048ca2d4cefeb13c14a89fc4bc833735f1c5e3a not found: ID does not exist" containerID="3f2b3aea7bd8cb2841e45336b048ca2d4cefeb13c14a89fc4bc833735f1c5e3a"
Oct 30 19:29:53.983828 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:53.983785    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2b3aea7bd8cb2841e45336b048ca2d4cefeb13c14a89fc4bc833735f1c5e3a"} err="failed to get container status \"3f2b3aea7bd8cb2841e45336b048ca2d4cefeb13c14a89fc4bc833735f1c5e3a\": rpc error: code = NotFound desc = could not find container \"3f2b3aea7bd8cb2841e45336b048ca2d4cefeb13c14a89fc4bc833735f1c5e3a\": container with ID starting with 3f2b3aea7bd8cb2841e45336b048ca2d4cefeb13c14a89fc4bc833735f1c5e3a not found: ID does not exist"
Oct 30 19:29:54.129733 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:54.129690    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"]
Oct 30 19:29:54.134417 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:54.134389    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824"]
Oct 30 19:29:54.663814 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:54.663769    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-7d30a0/simulator-single-main-7d30a0-kserve-5945d949bc-t8824" podUID="cc970b97-eec9-414e-8ac6-c36ea3e2f6ff" containerName="main" probeResult="failure" output="Get \"https://10.132.1.2:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:29:54.833414 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:54.833382    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63f3c6ad-febd-4b14-aaab-d73b7e8f53ba" path="/var/lib/kubelet/pods/63f3c6ad-febd-4b14-aaab-d73b7e8f53ba/volumes"
Oct 30 19:29:54.833844 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:29:54.833830    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc970b97-eec9-414e-8ac6-c36ea3e2f6ff" path="/var/lib/kubelet/pods/cc970b97-eec9-414e-8ac6-c36ea3e2f6ff/volumes"
Oct 30 19:30:00.165726 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:00.165693    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-26mm9"]
Oct 30 19:30:00.166226 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:00.166124    4924 cpu_manager.go:414] "containerMap: removing container" podUID="63f3c6ad-febd-4b14-aaab-d73b7e8f53ba" containerName="revoke-keys"
Oct 30 19:30:00.166226 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:00.166137    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63f3c6ad-febd-4b14-aaab-d73b7e8f53ba" containerName="revoke-keys"
Oct 30 19:30:00.166226 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:00.166155    4924 cpu_manager.go:414] "containerMap: removing container" podUID="cc970b97-eec9-414e-8ac6-c36ea3e2f6ff" containerName="main"
Oct 30 19:30:00.166226 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:00.166162    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc970b97-eec9-414e-8ac6-c36ea3e2f6ff" containerName="main"
Oct 30 19:30:00.166378 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:00.166236    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="63f3c6ad-febd-4b14-aaab-d73b7e8f53ba" containerName="revoke-keys"
Oct 30 19:30:00.166378 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:00.166247    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc970b97-eec9-414e-8ac6-c36ea3e2f6ff" containerName="main"
Oct 30 19:30:00.170502 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:00.170478    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-26mm9"
Oct 30 19:30:00.221085 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:00.221038    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38b8f8c8-85cc-4b2f-bed1-ee648966c13a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889810-26mm9\" (UID: \"38b8f8c8-85cc-4b2f-bed1-ee648966c13a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-26mm9"
Oct 30 19:30:00.221207 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:00.221158    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxdpg\" (UniqueName: \"kubernetes.io/projected/38b8f8c8-85cc-4b2f-bed1-ee648966c13a-kube-api-access-dxdpg\") pod \"maas-api-key-cleanup-29889810-26mm9\" (UID: \"38b8f8c8-85cc-4b2f-bed1-ee648966c13a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-26mm9"
Oct 30 19:30:00.224969 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:00.224945    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-26mm9"]
Oct 30 19:30:00.268862 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:00.268827    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:30:00.269089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:00.268900    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:30:00.269170 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:00.269156    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:30:00.322566 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:00.322531    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38b8f8c8-85cc-4b2f-bed1-ee648966c13a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889810-26mm9\" (UID: \"38b8f8c8-85cc-4b2f-bed1-ee648966c13a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-26mm9"
Oct 30 19:30:00.322716 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:00.322691    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxdpg\" (UniqueName: \"kubernetes.io/projected/38b8f8c8-85cc-4b2f-bed1-ee648966c13a-kube-api-access-dxdpg\") pod \"maas-api-key-cleanup-29889810-26mm9\" (UID: \"38b8f8c8-85cc-4b2f-bed1-ee648966c13a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-26mm9"
Oct 30 19:30:00.323193 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:00.323172    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38b8f8c8-85cc-4b2f-bed1-ee648966c13a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889810-26mm9\" (UID: \"38b8f8c8-85cc-4b2f-bed1-ee648966c13a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-26mm9"
Oct 30 19:30:00.378644 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:00.378616    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxdpg\" (UniqueName: \"kubernetes.io/projected/38b8f8c8-85cc-4b2f-bed1-ee648966c13a-kube-api-access-dxdpg\") pod \"maas-api-key-cleanup-29889810-26mm9\" (UID: \"38b8f8c8-85cc-4b2f-bed1-ee648966c13a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-26mm9"
Oct 30 19:30:00.483331 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:00.483268    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-26mm9"
Oct 30 19:30:00.851980 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:00.851956    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-26mm9"]
Oct 30 19:30:00.853772 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:30:00.853741    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38b8f8c8_85cc_4b2f_bed1_ee648966c13a.slice/crio-c462650ccba4ef7484ea23f18f0fb3e4455363d256de4ed7ee9ced29dabc18e8 WatchSource:0}: Error finding container c462650ccba4ef7484ea23f18f0fb3e4455363d256de4ed7ee9ced29dabc18e8: Status 404 returned error can't find the container with id c462650ccba4ef7484ea23f18f0fb3e4455363d256de4ed7ee9ced29dabc18e8
Oct 30 19:30:01.010408 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:01.010378    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-26mm9" event={"ID":"38b8f8c8-85cc-4b2f-bed1-ee648966c13a","Type":"ContainerStarted","Data":"c462650ccba4ef7484ea23f18f0fb3e4455363d256de4ed7ee9ced29dabc18e8"}
Oct 30 19:30:02.018579 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:02.018547    4924 generic.go:356] "Generic (PLEG): container finished" podID="38b8f8c8-85cc-4b2f-bed1-ee648966c13a" containerID="9a01d1b2737ce6f7a4a52cb7cdb5841821d6120d8e368deca739b2ea3ea9c63a" exitCode=60
Oct 30 19:30:02.018987 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:02.018639    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-26mm9" event={"ID":"38b8f8c8-85cc-4b2f-bed1-ee648966c13a","Type":"ContainerDied","Data":"9a01d1b2737ce6f7a4a52cb7cdb5841821d6120d8e368deca739b2ea3ea9c63a"}
Oct 30 19:30:02.018987 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:02.018871    4924 scope.go:122] "RemoveContainer" containerID="9a01d1b2737ce6f7a4a52cb7cdb5841821d6120d8e368deca739b2ea3ea9c63a"
Oct 30 19:30:02.485574 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:02.485532    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg" podUID="d3f85573-6434-4b07-b67e-396744b7d8bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:30:02.572087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:02.571984    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkbn4"]
Oct 30 19:30:02.574370 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:02.574341    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkbn4"]
Oct 30 19:30:02.834585 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:02.834509    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4b21c56-96fd-4356-b7b7-1c314e73ca63" path="/var/lib/kubelet/pods/a4b21c56-96fd-4356-b7b7-1c314e73ca63/volumes"
Oct 30 19:30:03.025706 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:03.025673    4924 generic.go:356] "Generic (PLEG): container finished" podID="38b8f8c8-85cc-4b2f-bed1-ee648966c13a" containerID="8d0683fa0d7ccdae8c4f1187c8ae4a60176e4b5a5083be3bf6e9de924adfc0a5" exitCode=60
Oct 30 19:30:03.026238 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:03.025751    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-26mm9" event={"ID":"38b8f8c8-85cc-4b2f-bed1-ee648966c13a","Type":"ContainerDied","Data":"8d0683fa0d7ccdae8c4f1187c8ae4a60176e4b5a5083be3bf6e9de924adfc0a5"}
Oct 30 19:30:03.026238 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:03.025808    4924 scope.go:122] "RemoveContainer" containerID="9a01d1b2737ce6f7a4a52cb7cdb5841821d6120d8e368deca739b2ea3ea9c63a"
Oct 30 19:30:03.026238 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:03.026100    4924 scope.go:122] "RemoveContainer" containerID="8d0683fa0d7ccdae8c4f1187c8ae4a60176e4b5a5083be3bf6e9de924adfc0a5"
Oct 30 19:30:03.026433 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:30:03.026412    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889810-26mm9_odh-ai-gateway-infra(38b8f8c8-85cc-4b2f-bed1-ee648966c13a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-26mm9" podUID="38b8f8c8-85cc-4b2f-bed1-ee648966c13a"
Oct 30 19:30:04.033420 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:04.033391    4924 scope.go:122] "RemoveContainer" containerID="8d0683fa0d7ccdae8c4f1187c8ae4a60176e4b5a5083be3bf6e9de924adfc0a5"
Oct 30 19:30:04.033829 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:30:04.033653    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889810-26mm9_odh-ai-gateway-infra(38b8f8c8-85cc-4b2f-bed1-ee648966c13a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-26mm9" podUID="38b8f8c8-85cc-4b2f-bed1-ee648966c13a"
Oct 30 19:30:10.374412 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:10.374342    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg"
Oct 30 19:30:10.523631 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:10.523596    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3f85573-6434-4b07-b67e-396744b7d8bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f85573-6434-4b07-b67e-396744b7d8bb-plugins-config-volume\") pod \"d3f85573-6434-4b07-b67e-396744b7d8bb\" (UID: \"d3f85573-6434-4b07-b67e-396744b7d8bb\") "
Oct 30 19:30:10.523815 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:10.523706    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3f85573-6434-4b07-b67e-396744b7d8bb-kube-api-access-g5cmq\" (UniqueName: \"kubernetes.io/projected/d3f85573-6434-4b07-b67e-396744b7d8bb-kube-api-access-g5cmq\") pod \"d3f85573-6434-4b07-b67e-396744b7d8bb\" (UID: \"d3f85573-6434-4b07-b67e-396744b7d8bb\") "
Oct 30 19:30:10.523984 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:10.523959    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f85573-6434-4b07-b67e-396744b7d8bb-plugins-config-volume" pod "d3f85573-6434-4b07-b67e-396744b7d8bb" (UID: "d3f85573-6434-4b07-b67e-396744b7d8bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:10.525768 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:10.525747    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f85573-6434-4b07-b67e-396744b7d8bb-kube-api-access-g5cmq" pod "d3f85573-6434-4b07-b67e-396744b7d8bb" (UID: "d3f85573-6434-4b07-b67e-396744b7d8bb"). InnerVolumeSpecName "kube-api-access-g5cmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:10.624759 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:10.624675    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5cmq\" (UniqueName: \"kubernetes.io/projected/d3f85573-6434-4b07-b67e-396744b7d8bb-kube-api-access-g5cmq\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:10.624759 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:10.624708    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f85573-6434-4b07-b67e-396744b7d8bb-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:11.072764 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:11.072655    4924 generic.go:356] "Generic (PLEG): container finished" podID="d3f85573-6434-4b07-b67e-396744b7d8bb" containerID="d6b2ed5feed56bbcbfa1b9fdbbd44c134fa460226ee1edb53d91dbcdd082ca39" exitCode=0
Oct 30 19:30:11.072764 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:11.072697    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg" event={"ID":"d3f85573-6434-4b07-b67e-396744b7d8bb","Type":"ContainerDied","Data":"d6b2ed5feed56bbcbfa1b9fdbbd44c134fa460226ee1edb53d91dbcdd082ca39"}
Oct 30 19:30:11.073007 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:11.072770    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg" event={"ID":"d3f85573-6434-4b07-b67e-396744b7d8bb","Type":"ContainerDied","Data":"2e97188d0116f0c16980c6f9f6b55934fa862dd7075a2da01578a5aea84147d9"}
Oct 30 19:30:11.073007 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:11.072709    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg"
Oct 30 19:30:11.073007 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:11.072803    4924 scope.go:122] "RemoveContainer" containerID="d6b2ed5feed56bbcbfa1b9fdbbd44c134fa460226ee1edb53d91dbcdd082ca39"
Oct 30 19:30:11.075334 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:11.075310    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rlwcq"]
Oct 30 19:30:11.082174 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:11.082156    4924 scope.go:122] "RemoveContainer" containerID="d6b2ed5feed56bbcbfa1b9fdbbd44c134fa460226ee1edb53d91dbcdd082ca39"
Oct 30 19:30:11.082439 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:30:11.082420    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6b2ed5feed56bbcbfa1b9fdbbd44c134fa460226ee1edb53d91dbcdd082ca39\": container with ID starting with d6b2ed5feed56bbcbfa1b9fdbbd44c134fa460226ee1edb53d91dbcdd082ca39 not found: ID does not exist" containerID="d6b2ed5feed56bbcbfa1b9fdbbd44c134fa460226ee1edb53d91dbcdd082ca39"
Oct 30 19:30:11.082501 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:11.082446    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b2ed5feed56bbcbfa1b9fdbbd44c134fa460226ee1edb53d91dbcdd082ca39"} err="failed to get container status \"d6b2ed5feed56bbcbfa1b9fdbbd44c134fa460226ee1edb53d91dbcdd082ca39\": rpc error: code = NotFound desc = could not find container \"d6b2ed5feed56bbcbfa1b9fdbbd44c134fa460226ee1edb53d91dbcdd082ca39\": container with ID starting with d6b2ed5feed56bbcbfa1b9fdbbd44c134fa460226ee1edb53d91dbcdd082ca39 not found: ID does not exist"
Oct 30 19:30:11.121803 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:11.121766    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rlwcq"]
Oct 30 19:30:11.231248 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:11.231213    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg"]
Oct 30 19:30:11.248258 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:11.248229    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7d30a0-7ddf4fb78krpzg"]
Oct 30 19:30:12.833640 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:12.833609    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77e1894e-9c24-4c9a-ba79-5626c51a9baf" path="/var/lib/kubelet/pods/77e1894e-9c24-4c9a-ba79-5626c51a9baf/volumes"
Oct 30 19:30:12.834036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:12.833990    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3f85573-6434-4b07-b67e-396744b7d8bb" path="/var/lib/kubelet/pods/d3f85573-6434-4b07-b67e-396744b7d8bb/volumes"
Oct 30 19:30:15.710487 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:30:15.710451    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:30:17.827491 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:17.827459    4924 scope.go:122] "RemoveContainer" containerID="8d0683fa0d7ccdae8c4f1187c8ae4a60176e4b5a5083be3bf6e9de924adfc0a5"
Oct 30 19:30:19.115111 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:19.115076    4924 generic.go:356] "Generic (PLEG): container finished" podID="38b8f8c8-85cc-4b2f-bed1-ee648966c13a" containerID="41117fa6d5575e5e344ffff0cb687e28bb400d0e255412c1f759d670e6bf61fc" exitCode=60
Oct 30 19:30:19.115582 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:19.115132    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-26mm9" event={"ID":"38b8f8c8-85cc-4b2f-bed1-ee648966c13a","Type":"ContainerDied","Data":"41117fa6d5575e5e344ffff0cb687e28bb400d0e255412c1f759d670e6bf61fc"}
Oct 30 19:30:19.115582 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:19.115189    4924 scope.go:122] "RemoveContainer" containerID="8d0683fa0d7ccdae8c4f1187c8ae4a60176e4b5a5083be3bf6e9de924adfc0a5"
Oct 30 19:30:19.115582 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:19.115540    4924 scope.go:122] "RemoveContainer" containerID="41117fa6d5575e5e344ffff0cb687e28bb400d0e255412c1f759d670e6bf61fc"
Oct 30 19:30:19.115827 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:30:19.115804    4924 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889810-26mm9_odh-ai-gateway-infra(38b8f8c8-85cc-4b2f-bed1-ee648966c13a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-26mm9" podUID="38b8f8c8-85cc-4b2f-bed1-ee648966c13a"
Oct 30 19:30:20.241142 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:20.241103    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-26mm9"]
Oct 30 19:30:21.012281 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:21.012248    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-26mm9"
Oct 30 19:30:21.139298 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:21.138241    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38b8f8c8-85cc-4b2f-bed1-ee648966c13a-kube-api-access-dxdpg\" (UniqueName: \"kubernetes.io/projected/38b8f8c8-85cc-4b2f-bed1-ee648966c13a-kube-api-access-dxdpg\") pod \"38b8f8c8-85cc-4b2f-bed1-ee648966c13a\" (UID: \"38b8f8c8-85cc-4b2f-bed1-ee648966c13a\") "
Oct 30 19:30:21.139298 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:21.138315    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38b8f8c8-85cc-4b2f-bed1-ee648966c13a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38b8f8c8-85cc-4b2f-bed1-ee648966c13a-maas-api-service-ca\") pod \"38b8f8c8-85cc-4b2f-bed1-ee648966c13a\" (UID: \"38b8f8c8-85cc-4b2f-bed1-ee648966c13a\") "
Oct 30 19:30:21.139808 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:21.139762    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38b8f8c8-85cc-4b2f-bed1-ee648966c13a-maas-api-service-ca" pod "38b8f8c8-85cc-4b2f-bed1-ee648966c13a" (UID: "38b8f8c8-85cc-4b2f-bed1-ee648966c13a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:21.162333 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:21.162252    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-26mm9" event={"ID":"38b8f8c8-85cc-4b2f-bed1-ee648966c13a","Type":"ContainerDied","Data":"c462650ccba4ef7484ea23f18f0fb3e4455363d256de4ed7ee9ced29dabc18e8"}
Oct 30 19:30:21.162578 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:21.162545    4924 scope.go:122] "RemoveContainer" containerID="41117fa6d5575e5e344ffff0cb687e28bb400d0e255412c1f759d670e6bf61fc"
Oct 30 19:30:21.162969 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:21.162765    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-26mm9"
Oct 30 19:30:21.168154 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:21.168096    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38b8f8c8-85cc-4b2f-bed1-ee648966c13a-kube-api-access-dxdpg" pod "38b8f8c8-85cc-4b2f-bed1-ee648966c13a" (UID: "38b8f8c8-85cc-4b2f-bed1-ee648966c13a"). InnerVolumeSpecName "kube-api-access-dxdpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:21.240984 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:21.240381    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxdpg\" (UniqueName: \"kubernetes.io/projected/38b8f8c8-85cc-4b2f-bed1-ee648966c13a-kube-api-access-dxdpg\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:21.240984 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:21.240418    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38b8f8c8-85cc-4b2f-bed1-ee648966c13a-maas-api-service-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:21.579925 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:21.579890    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-26mm9"]
Oct 30 19:30:21.626701 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:21.626666    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-26mm9"]
Oct 30 19:30:22.833961 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:22.833921    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38b8f8c8-85cc-4b2f-bed1-ee648966c13a" path="/var/lib/kubelet/pods/38b8f8c8-85cc-4b2f-bed1-ee648966c13a/volumes"
Oct 30 19:30:39.055498 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:39.055464    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6xlph"]
Oct 30 19:30:39.102863 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:39.102827    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6xlph"]
Oct 30 19:30:40.832845 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:40.832810    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd626f9-6b88-4e6d-8b31-21670e7827ab" path="/var/lib/kubelet/pods/abd626f9-6b88-4e6d-8b31-21670e7827ab/volumes"
Oct 30 19:30:41.904314 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:41.904271    4924 scope.go:122] "RemoveContainer" containerID="a9f027f4671fec8212850af88aae18090387292621684068306851136d6d209d"
Oct 30 19:30:41.913033 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:41.913008    4924 scope.go:122] "RemoveContainer" containerID="4161d9f643679aea61d899728df1009d54676767b0c17d9c8b45fa03af22336f"
Oct 30 19:30:41.920622 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:30:41.920602    4924 scope.go:122] "RemoveContainer" containerID="d56e9ba53ba3607600d7b14868bb25ebc6ee8dff23fcf311c7105c9589f3d428"
Oct 30 19:30:45.710430 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:30:45.710399    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:31:09.046890 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:31:09.046854    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jk7f9"]
Oct 30 19:31:09.093578 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:31:09.093547    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jk7f9"]
Oct 30 19:31:10.833678 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:31:10.833642    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dee08af-8e31-40cc-8913-99f389aeb86c" path="/var/lib/kubelet/pods/8dee08af-8e31-40cc-8913-99f389aeb86c/volumes"
Oct 30 19:31:15.711344 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:31:15.711310    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:31:30.047134 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:31:30.047042    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f7g99"]
Oct 30 19:31:30.093821 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:31:30.093784    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-f7g99"]
Oct 30 19:31:30.833807 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:31:30.833773    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecbc2b45-09db-4770-8490-af7a15ccac8c" path="/var/lib/kubelet/pods/ecbc2b45-09db-4770-8490-af7a15ccac8c/volumes"
Oct 30 19:31:39.383520 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:31:39.383485    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 19:31:39.389039 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:31:39.389011    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 19:31:41.960663 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:31:41.960627    4924 scope.go:122] "RemoveContainer" containerID="27e0026130685afb0712d6a508addcdb02360e3cc2519d2c13d596c2b669386e"
Oct 30 19:31:41.969411 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:31:41.969389    4924 scope.go:122] "RemoveContainer" containerID="84b1b8201da1bdc5f4292e70fcdde05c1d806788a35db851faf45e8ef64589d2"
Oct 30 19:31:45.710517 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:31:45.710479    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:32:05.759364 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.759326    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tfrdq"]
Oct 30 19:32:05.759779 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.759732    4924 cpu_manager.go:414] "containerMap: removing container" podUID="d3f85573-6434-4b07-b67e-396744b7d8bb" containerName="payload-processing"
Oct 30 19:32:05.759779 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.759745    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f85573-6434-4b07-b67e-396744b7d8bb" containerName="payload-processing"
Oct 30 19:32:05.759779 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.759766    4924 cpu_manager.go:414] "containerMap: removing container" podUID="38b8f8c8-85cc-4b2f-bed1-ee648966c13a" containerName="cleanup"
Oct 30 19:32:05.759779 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.759774    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38b8f8c8-85cc-4b2f-bed1-ee648966c13a" containerName="cleanup"
Oct 30 19:32:05.759918 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.759799    4924 cpu_manager.go:414] "containerMap: removing container" podUID="38b8f8c8-85cc-4b2f-bed1-ee648966c13a" containerName="cleanup"
Oct 30 19:32:05.759918 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.759804    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38b8f8c8-85cc-4b2f-bed1-ee648966c13a" containerName="cleanup"
Oct 30 19:32:05.759918 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.759813    4924 cpu_manager.go:414] "containerMap: removing container" podUID="38b8f8c8-85cc-4b2f-bed1-ee648966c13a" containerName="cleanup"
Oct 30 19:32:05.759918 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.759817    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38b8f8c8-85cc-4b2f-bed1-ee648966c13a" containerName="cleanup"
Oct 30 19:32:05.759918 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.759878    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="38b8f8c8-85cc-4b2f-bed1-ee648966c13a" containerName="cleanup"
Oct 30 19:32:05.759918 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.759887    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="38b8f8c8-85cc-4b2f-bed1-ee648966c13a" containerName="cleanup"
Oct 30 19:32:05.759918 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.759896    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f85573-6434-4b07-b67e-396744b7d8bb" containerName="payload-processing"
Oct 30 19:32:05.764610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.764590    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tfrdq"
Oct 30 19:32:05.816611 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.816577    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:32:05.819015 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.818989    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tfrdq"]
Oct 30 19:32:05.840350 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.840316    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c46b0dc6-fa8d-468b-a7dc-49bdbb88670c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-tfrdq\" (UID: \"c46b0dc6-fa8d-468b-a7dc-49bdbb88670c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tfrdq"
Oct 30 19:32:05.840524 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.840437    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44nwr\" (UniqueName: \"kubernetes.io/projected/c46b0dc6-fa8d-468b-a7dc-49bdbb88670c-kube-api-access-44nwr\") pod \"maas-api-revoke-sub-d576e68b-tfrdq\" (UID: \"c46b0dc6-fa8d-468b-a7dc-49bdbb88670c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tfrdq"
Oct 30 19:32:05.840524 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.840477    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c46b0dc6-fa8d-468b-a7dc-49bdbb88670c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-tfrdq\" (UID: \"c46b0dc6-fa8d-468b-a7dc-49bdbb88670c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tfrdq"
Oct 30 19:32:05.862822 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.862784    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:32:05.863367 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.863348    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:32:05.863668 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.863645    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:32:05.941055 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.941016    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c46b0dc6-fa8d-468b-a7dc-49bdbb88670c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-tfrdq\" (UID: \"c46b0dc6-fa8d-468b-a7dc-49bdbb88670c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tfrdq"
Oct 30 19:32:05.941288 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.941123    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44nwr\" (UniqueName: \"kubernetes.io/projected/c46b0dc6-fa8d-468b-a7dc-49bdbb88670c-kube-api-access-44nwr\") pod \"maas-api-revoke-sub-d576e68b-tfrdq\" (UID: \"c46b0dc6-fa8d-468b-a7dc-49bdbb88670c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tfrdq"
Oct 30 19:32:05.941288 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.941150    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c46b0dc6-fa8d-468b-a7dc-49bdbb88670c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-tfrdq\" (UID: \"c46b0dc6-fa8d-468b-a7dc-49bdbb88670c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tfrdq"
Oct 30 19:32:05.941821 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.941800    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c46b0dc6-fa8d-468b-a7dc-49bdbb88670c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-tfrdq\" (UID: \"c46b0dc6-fa8d-468b-a7dc-49bdbb88670c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tfrdq"
Oct 30 19:32:05.943477 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.943454    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c46b0dc6-fa8d-468b-a7dc-49bdbb88670c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-tfrdq\" (UID: \"c46b0dc6-fa8d-468b-a7dc-49bdbb88670c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tfrdq"
Oct 30 19:32:05.996691 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:05.996654    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44nwr\" (UniqueName: \"kubernetes.io/projected/c46b0dc6-fa8d-468b-a7dc-49bdbb88670c-kube-api-access-44nwr\") pod \"maas-api-revoke-sub-d576e68b-tfrdq\" (UID: \"c46b0dc6-fa8d-468b-a7dc-49bdbb88670c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tfrdq"
Oct 30 19:32:06.079294 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:06.079204    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tfrdq"
Oct 30 19:32:06.444365 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:06.444333    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tfrdq"]
Oct 30 19:32:06.446164 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:32:06.446129    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc46b0dc6_fa8d_468b_a7dc_49bdbb88670c.slice/crio-fac534e9db093e1d81beff29a0083af5be8e4ea42d788142871a2666fbb49e04 WatchSource:0}: Error finding container fac534e9db093e1d81beff29a0083af5be8e4ea42d788142871a2666fbb49e04: Status 404 returned error can't find the container with id fac534e9db093e1d81beff29a0083af5be8e4ea42d788142871a2666fbb49e04
Oct 30 19:32:06.715257 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:06.715230    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tfrdq" event={"ID":"c46b0dc6-fa8d-468b-a7dc-49bdbb88670c","Type":"ContainerStarted","Data":"fac534e9db093e1d81beff29a0083af5be8e4ea42d788142871a2666fbb49e04"}
Oct 30 19:32:07.722194 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:07.722156    4924 generic.go:356] "Generic (PLEG): container finished" podID="c46b0dc6-fa8d-468b-a7dc-49bdbb88670c" containerID="cdd56da3d5e25b0d8900ccd192fb053d04a7417d07d7a6c755e2f9697fe84310" exitCode=0
Oct 30 19:32:07.722194 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:07.722200    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tfrdq" event={"ID":"c46b0dc6-fa8d-468b-a7dc-49bdbb88670c","Type":"ContainerDied","Data":"cdd56da3d5e25b0d8900ccd192fb053d04a7417d07d7a6c755e2f9697fe84310"}
Oct 30 19:32:08.863174 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:08.863148    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tfrdq"
Oct 30 19:32:08.971696 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:08.971658    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c46b0dc6-fa8d-468b-a7dc-49bdbb88670c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c46b0dc6-fa8d-468b-a7dc-49bdbb88670c-maas-api-openshift-ca\") pod \"c46b0dc6-fa8d-468b-a7dc-49bdbb88670c\" (UID: \"c46b0dc6-fa8d-468b-a7dc-49bdbb88670c\") "
Oct 30 19:32:08.971696 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:08.971701    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c46b0dc6-fa8d-468b-a7dc-49bdbb88670c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c46b0dc6-fa8d-468b-a7dc-49bdbb88670c-maas-api-xks-ca\") pod \"c46b0dc6-fa8d-468b-a7dc-49bdbb88670c\" (UID: \"c46b0dc6-fa8d-468b-a7dc-49bdbb88670c\") "
Oct 30 19:32:08.971980 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:08.971723    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c46b0dc6-fa8d-468b-a7dc-49bdbb88670c-kube-api-access-44nwr\" (UniqueName: \"kubernetes.io/projected/c46b0dc6-fa8d-468b-a7dc-49bdbb88670c-kube-api-access-44nwr\") pod \"c46b0dc6-fa8d-468b-a7dc-49bdbb88670c\" (UID: \"c46b0dc6-fa8d-468b-a7dc-49bdbb88670c\") "
Oct 30 19:32:08.972177 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:08.972147    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c46b0dc6-fa8d-468b-a7dc-49bdbb88670c-maas-api-openshift-ca" pod "c46b0dc6-fa8d-468b-a7dc-49bdbb88670c" (UID: "c46b0dc6-fa8d-468b-a7dc-49bdbb88670c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:32:08.973954 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:08.973917    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c46b0dc6-fa8d-468b-a7dc-49bdbb88670c-maas-api-xks-ca" pod "c46b0dc6-fa8d-468b-a7dc-49bdbb88670c" (UID: "c46b0dc6-fa8d-468b-a7dc-49bdbb88670c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:32:08.974056 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:08.973968    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c46b0dc6-fa8d-468b-a7dc-49bdbb88670c-kube-api-access-44nwr" pod "c46b0dc6-fa8d-468b-a7dc-49bdbb88670c" (UID: "c46b0dc6-fa8d-468b-a7dc-49bdbb88670c"). InnerVolumeSpecName "kube-api-access-44nwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:09.073203 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:09.073165    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c46b0dc6-fa8d-468b-a7dc-49bdbb88670c-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:09.073203 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:09.073199    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c46b0dc6-fa8d-468b-a7dc-49bdbb88670c-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:09.073203 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:09.073211    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44nwr\" (UniqueName: \"kubernetes.io/projected/c46b0dc6-fa8d-468b-a7dc-49bdbb88670c-kube-api-access-44nwr\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:09.733775 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:09.733740    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tfrdq"
Oct 30 19:32:09.733996 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:09.733769    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tfrdq" event={"ID":"c46b0dc6-fa8d-468b-a7dc-49bdbb88670c","Type":"ContainerDied","Data":"fac534e9db093e1d81beff29a0083af5be8e4ea42d788142871a2666fbb49e04"}
Oct 30 19:32:09.733996 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:09.733823    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fac534e9db093e1d81beff29a0083af5be8e4ea42d788142871a2666fbb49e04"
Oct 30 19:32:15.711282 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:32:15.711237    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:32:27.262668 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:27.262626    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lq6dq"]
Oct 30 19:32:27.263249 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:27.263228    4924 cpu_manager.go:414] "containerMap: removing container" podUID="c46b0dc6-fa8d-468b-a7dc-49bdbb88670c" containerName="revoke-keys"
Oct 30 19:32:27.263337 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:27.263252    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c46b0dc6-fa8d-468b-a7dc-49bdbb88670c" containerName="revoke-keys"
Oct 30 19:32:27.263397 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:27.263370    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="38b8f8c8-85cc-4b2f-bed1-ee648966c13a" containerName="cleanup"
Oct 30 19:32:27.263397 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:27.263386    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="c46b0dc6-fa8d-468b-a7dc-49bdbb88670c" containerName="revoke-keys"
Oct 30 19:32:27.266781 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:27.266759    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lq6dq"
Oct 30 19:32:27.318885 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:27.318855    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:32:27.364876 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:27.364841    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:32:27.365115 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:27.364887    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:32:27.365262 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:27.365245    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:32:27.366836 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:27.366810    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lq6dq"]
Oct 30 19:32:27.394849 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:27.394825    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96237fa-3d73-4acc-bef1-35a49c783d20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lq6dq\" (UID: \"a96237fa-3d73-4acc-bef1-35a49c783d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lq6dq"
Oct 30 19:32:27.394952 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:27.394886    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn7rd\" (UniqueName: \"kubernetes.io/projected/a96237fa-3d73-4acc-bef1-35a49c783d20-kube-api-access-wn7rd\") pod \"maas-api-revoke-sub-d2bc93bc-lq6dq\" (UID: \"a96237fa-3d73-4acc-bef1-35a49c783d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lq6dq"
Oct 30 19:32:27.395030 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:27.395010    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96237fa-3d73-4acc-bef1-35a49c783d20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lq6dq\" (UID: \"a96237fa-3d73-4acc-bef1-35a49c783d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lq6dq"
Oct 30 19:32:27.495898 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:27.495868    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96237fa-3d73-4acc-bef1-35a49c783d20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lq6dq\" (UID: \"a96237fa-3d73-4acc-bef1-35a49c783d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lq6dq"
Oct 30 19:32:27.496124 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:27.495940    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn7rd\" (UniqueName: \"kubernetes.io/projected/a96237fa-3d73-4acc-bef1-35a49c783d20-kube-api-access-wn7rd\") pod \"maas-api-revoke-sub-d2bc93bc-lq6dq\" (UID: \"a96237fa-3d73-4acc-bef1-35a49c783d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lq6dq"
Oct 30 19:32:27.496124 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:27.495974    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96237fa-3d73-4acc-bef1-35a49c783d20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lq6dq\" (UID: \"a96237fa-3d73-4acc-bef1-35a49c783d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lq6dq"
Oct 30 19:32:27.496698 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:27.496675    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96237fa-3d73-4acc-bef1-35a49c783d20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lq6dq\" (UID: \"a96237fa-3d73-4acc-bef1-35a49c783d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lq6dq"
Oct 30 19:32:27.498202 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:27.498181    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96237fa-3d73-4acc-bef1-35a49c783d20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lq6dq\" (UID: \"a96237fa-3d73-4acc-bef1-35a49c783d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lq6dq"
Oct 30 19:32:27.551207 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:27.551141    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn7rd\" (UniqueName: \"kubernetes.io/projected/a96237fa-3d73-4acc-bef1-35a49c783d20-kube-api-access-wn7rd\") pod \"maas-api-revoke-sub-d2bc93bc-lq6dq\" (UID: \"a96237fa-3d73-4acc-bef1-35a49c783d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lq6dq"
Oct 30 19:32:27.579985 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:27.579952    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lq6dq"
Oct 30 19:32:27.944723 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:27.944698    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lq6dq"]
Oct 30 19:32:27.947932 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:32:27.947899    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96237fa_3d73_4acc_bef1_35a49c783d20.slice/crio-8e9941f534b6324caf7da01505d269d20133a50e55785e1a736d7fb4120721ba WatchSource:0}: Error finding container 8e9941f534b6324caf7da01505d269d20133a50e55785e1a736d7fb4120721ba: Status 404 returned error can't find the container with id 8e9941f534b6324caf7da01505d269d20133a50e55785e1a736d7fb4120721ba
Oct 30 19:32:28.860948 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:28.860910    4924 generic.go:356] "Generic (PLEG): container finished" podID="a96237fa-3d73-4acc-bef1-35a49c783d20" containerID="fab137135ed32d5c0f0edd15cbd5974de1806f06553535ac3120bff993388cf6" exitCode=0
Oct 30 19:32:28.861446 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:28.860999    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lq6dq" event={"ID":"a96237fa-3d73-4acc-bef1-35a49c783d20","Type":"ContainerDied","Data":"fab137135ed32d5c0f0edd15cbd5974de1806f06553535ac3120bff993388cf6"}
Oct 30 19:32:28.861446 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:28.861047    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lq6dq" event={"ID":"a96237fa-3d73-4acc-bef1-35a49c783d20","Type":"ContainerStarted","Data":"8e9941f534b6324caf7da01505d269d20133a50e55785e1a736d7fb4120721ba"}
Oct 30 19:32:29.997809 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:29.997787    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lq6dq"
Oct 30 19:32:30.121032 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:30.120963    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a96237fa-3d73-4acc-bef1-35a49c783d20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96237fa-3d73-4acc-bef1-35a49c783d20-maas-api-openshift-ca\") pod \"a96237fa-3d73-4acc-bef1-35a49c783d20\" (UID: \"a96237fa-3d73-4acc-bef1-35a49c783d20\") "
Oct 30 19:32:30.121032 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:30.121016    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a96237fa-3d73-4acc-bef1-35a49c783d20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96237fa-3d73-4acc-bef1-35a49c783d20-maas-api-xks-ca\") pod \"a96237fa-3d73-4acc-bef1-35a49c783d20\" (UID: \"a96237fa-3d73-4acc-bef1-35a49c783d20\") "
Oct 30 19:32:30.121238 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:30.121180    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a96237fa-3d73-4acc-bef1-35a49c783d20-kube-api-access-wn7rd\" (UniqueName: \"kubernetes.io/projected/a96237fa-3d73-4acc-bef1-35a49c783d20-kube-api-access-wn7rd\") pod \"a96237fa-3d73-4acc-bef1-35a49c783d20\" (UID: \"a96237fa-3d73-4acc-bef1-35a49c783d20\") "
Oct 30 19:32:30.121442 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:30.121415    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a96237fa-3d73-4acc-bef1-35a49c783d20-maas-api-openshift-ca" pod "a96237fa-3d73-4acc-bef1-35a49c783d20" (UID: "a96237fa-3d73-4acc-bef1-35a49c783d20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:32:30.121540 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:30.121518    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96237fa-3d73-4acc-bef1-35a49c783d20-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:30.123237 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:30.123213    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a96237fa-3d73-4acc-bef1-35a49c783d20-maas-api-xks-ca" pod "a96237fa-3d73-4acc-bef1-35a49c783d20" (UID: "a96237fa-3d73-4acc-bef1-35a49c783d20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:32:30.123237 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:30.123219    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96237fa-3d73-4acc-bef1-35a49c783d20-kube-api-access-wn7rd" pod "a96237fa-3d73-4acc-bef1-35a49c783d20" (UID: "a96237fa-3d73-4acc-bef1-35a49c783d20"). InnerVolumeSpecName "kube-api-access-wn7rd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:30.222020 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:30.221985    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96237fa-3d73-4acc-bef1-35a49c783d20-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:30.222020 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:30.222015    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn7rd\" (UniqueName: \"kubernetes.io/projected/a96237fa-3d73-4acc-bef1-35a49c783d20-kube-api-access-wn7rd\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:30.872270 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:30.872238    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lq6dq"
Oct 30 19:32:30.872447 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:30.872275    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lq6dq" event={"ID":"a96237fa-3d73-4acc-bef1-35a49c783d20","Type":"ContainerDied","Data":"8e9941f534b6324caf7da01505d269d20133a50e55785e1a736d7fb4120721ba"}
Oct 30 19:32:30.872447 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:30.872320    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e9941f534b6324caf7da01505d269d20133a50e55785e1a736d7fb4120721ba"
Oct 30 19:32:41.992696 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:41.992664    4924 scope.go:122] "RemoveContainer" containerID="cd02db617cb3a4653872c2885f2fac88f6973b313e45ba5e9f202512178326df"
Oct 30 19:32:42.001643 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:42.001623    4924 scope.go:122] "RemoveContainer" containerID="d1ea99f5f29039a740b7bb4b4277b487fa77474303d6a3b12e2b9a78672bd4cc"
Oct 30 19:32:42.009701 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:42.009683    4924 scope.go:122] "RemoveContainer" containerID="f0380d702a1adf2a45e5e5cd3a7b663a3e53c147a8fa3fe73a315a29716ae5d4"
Oct 30 19:32:45.711627 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:32:45.711594    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:32:47.550964 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:47.550930    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk"]
Oct 30 19:32:47.551547 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:47.551517    4924 cpu_manager.go:414] "containerMap: removing container" podUID="a96237fa-3d73-4acc-bef1-35a49c783d20" containerName="revoke-keys"
Oct 30 19:32:47.551547 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:47.551544    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a96237fa-3d73-4acc-bef1-35a49c783d20" containerName="revoke-keys"
Oct 30 19:32:47.551779 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:47.551633    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="a96237fa-3d73-4acc-bef1-35a49c783d20" containerName="revoke-keys"
Oct 30 19:32:47.556240 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:47.556223    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk"
Oct 30 19:32:47.588398 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:47.588373    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt987\" (UniqueName: \"kubernetes.io/projected/135a7159-612b-432b-9010-dcc1a0a0e071-kube-api-access-jt987\") pod \"payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk\" (UID: \"135a7159-612b-432b-9010-dcc1a0a0e071\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk"
Oct 30 19:32:47.588518 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:47.588436    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/135a7159-612b-432b-9010-dcc1a0a0e071-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk\" (UID: \"135a7159-612b-432b-9010-dcc1a0a0e071\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk"
Oct 30 19:32:47.611547 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:47.611526    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-34d64c\""
Oct 30 19:32:47.657851 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:47.657815    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-97f4d5f1-dockercfg-cfj98\""
Oct 30 19:32:47.660240 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:47.660217    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk"]
Oct 30 19:32:47.689241 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:47.689212    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt987\" (UniqueName: \"kubernetes.io/projected/135a7159-612b-432b-9010-dcc1a0a0e071-kube-api-access-jt987\") pod \"payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk\" (UID: \"135a7159-612b-432b-9010-dcc1a0a0e071\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk"
Oct 30 19:32:47.689391 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:47.689275    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/135a7159-612b-432b-9010-dcc1a0a0e071-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk\" (UID: \"135a7159-612b-432b-9010-dcc1a0a0e071\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk"
Oct 30 19:32:47.689967 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:47.689948    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/135a7159-612b-432b-9010-dcc1a0a0e071-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk\" (UID: \"135a7159-612b-432b-9010-dcc1a0a0e071\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk"
Oct 30 19:32:47.746369 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:47.746328    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt987\" (UniqueName: \"kubernetes.io/projected/135a7159-612b-432b-9010-dcc1a0a0e071-kube-api-access-jt987\") pod \"payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk\" (UID: \"135a7159-612b-432b-9010-dcc1a0a0e071\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk"
Oct 30 19:32:47.869409 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:47.869377    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk"
Oct 30 19:32:48.232935 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:48.232909    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk"]
Oct 30 19:32:48.234954 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:32:48.234924    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135a7159_612b_432b_9010_dcc1a0a0e071.slice/crio-15af697c5869270510f24842dfd9dd927a55b971bd094045e51ce5a8f0c0af7d WatchSource:0}: Error finding container 15af697c5869270510f24842dfd9dd927a55b971bd094045e51ce5a8f0c0af7d: Status 404 returned error can't find the container with id 15af697c5869270510f24842dfd9dd927a55b971bd094045e51ce5a8f0c0af7d
Oct 30 19:32:48.969595 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:48.969554    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk" event={"ID":"135a7159-612b-432b-9010-dcc1a0a0e071","Type":"ContainerStarted","Data":"e2d8d23aa59dc47ea219bc3d5b909c167d4396b509d2ca6bde5f7da75348722a"}
Oct 30 19:32:48.970092 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:48.969615    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk" event={"ID":"135a7159-612b-432b-9010-dcc1a0a0e071","Type":"ContainerStarted","Data":"15af697c5869270510f24842dfd9dd927a55b971bd094045e51ce5a8f0c0af7d"}
Oct 30 19:32:48.970092 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:48.969727    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk"
Oct 30 19:32:49.079464 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:49.079408    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk" podStartSLOduration=2.079396502 podStartE2EDuration="2.079396502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:49.079194791 +0000 UTC m=+3370.356245236" watchObservedRunningTime="2026-10-30 19:32:49.079396502 +0000 UTC m=+3370.356446944"
Oct 30 19:32:59.978019 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:32:59.977938    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk"
Oct 30 19:33:11.862558 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:11.862522    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"]
Oct 30 19:33:11.866252 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:11.866231    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"
Oct 30 19:33:11.939741 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:11.939707    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftm9q\" (UniqueName: \"kubernetes.io/projected/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-kube-api-access-ftm9q\") pod \"simulator-single-main-34d64c-kserve-55b74b76fd-676j9\" (UID: \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\") " pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"
Oct 30 19:33:11.939909 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:11.939747    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-tls-certs\") pod \"simulator-single-main-34d64c-kserve-55b74b76fd-676j9\" (UID: \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\") " pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"
Oct 30 19:33:11.939909 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:11.939779    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-dshm\") pod \"simulator-single-main-34d64c-kserve-55b74b76fd-676j9\" (UID: \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\") " pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"
Oct 30 19:33:11.939993 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:11.939932    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-home\") pod \"simulator-single-main-34d64c-kserve-55b74b76fd-676j9\" (UID: \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\") " pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"
Oct 30 19:33:11.939993 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:11.939964    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-model-cache\") pod \"simulator-single-main-34d64c-kserve-55b74b76fd-676j9\" (UID: \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\") " pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"
Oct 30 19:33:11.939993 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:11.939990    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-tmp-dir\") pod \"simulator-single-main-34d64c-kserve-55b74b76fd-676j9\" (UID: \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\") " pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"
Oct 30 19:33:11.963303 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:11.963271    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-34d64c\"/\"openshift-service-ca.crt\""
Oct 30 19:33:11.963457 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:11.963330    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-34d64c\"/\"default-dockercfg-f5d7x\""
Oct 30 19:33:11.963457 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:11.963381    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-34d64c\"/\"simulator-single-main-34d64c-kserve-self-signed-certs\""
Oct 30 19:33:11.963538 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:11.963488    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-34d64c\"/\"kube-root-ca.crt\""
Oct 30 19:33:11.965839 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:11.965820    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"]
Oct 30 19:33:12.040681 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:12.040640    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftm9q\" (UniqueName: \"kubernetes.io/projected/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-kube-api-access-ftm9q\") pod \"simulator-single-main-34d64c-kserve-55b74b76fd-676j9\" (UID: \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\") " pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"
Oct 30 19:33:12.040681 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:12.040686    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-tls-certs\") pod \"simulator-single-main-34d64c-kserve-55b74b76fd-676j9\" (UID: \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\") " pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"
Oct 30 19:33:12.040990 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:12.040719    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-dshm\") pod \"simulator-single-main-34d64c-kserve-55b74b76fd-676j9\" (UID: \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\") " pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"
Oct 30 19:33:12.040990 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:12.040764    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-home\") pod \"simulator-single-main-34d64c-kserve-55b74b76fd-676j9\" (UID: \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\") " pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"
Oct 30 19:33:12.040990 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:12.040901    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-model-cache\") pod \"simulator-single-main-34d64c-kserve-55b74b76fd-676j9\" (UID: \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\") " pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"
Oct 30 19:33:12.040990 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:12.040956    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-tmp-dir\") pod \"simulator-single-main-34d64c-kserve-55b74b76fd-676j9\" (UID: \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\") " pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"
Oct 30 19:33:12.041268 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:12.041241    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-home\") pod \"simulator-single-main-34d64c-kserve-55b74b76fd-676j9\" (UID: \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\") " pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"
Oct 30 19:33:12.041323 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:12.041299    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-tmp-dir\") pod \"simulator-single-main-34d64c-kserve-55b74b76fd-676j9\" (UID: \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\") " pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"
Oct 30 19:33:12.041414 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:12.041394    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-model-cache\") pod \"simulator-single-main-34d64c-kserve-55b74b76fd-676j9\" (UID: \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\") " pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"
Oct 30 19:33:12.043128 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:12.043107    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-dshm\") pod \"simulator-single-main-34d64c-kserve-55b74b76fd-676j9\" (UID: \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\") " pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"
Oct 30 19:33:12.043408 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:12.043391    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-tls-certs\") pod \"simulator-single-main-34d64c-kserve-55b74b76fd-676j9\" (UID: \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\") " pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"
Oct 30 19:33:12.098638 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:12.098605    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftm9q\" (UniqueName: \"kubernetes.io/projected/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-kube-api-access-ftm9q\") pod \"simulator-single-main-34d64c-kserve-55b74b76fd-676j9\" (UID: \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\") " pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"
Oct 30 19:33:12.177709 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:12.177606    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"
Oct 30 19:33:12.559210 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:12.559185    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"]
Oct 30 19:33:12.562145 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:33:12.562119    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd12458e_1fe1_43cf_af8d_cd6c43473c7e.slice/crio-575036b6eeb49155f8b9e43a901d5ee22f05275c5d6521d500d34d07b3378c2b WatchSource:0}: Error finding container 575036b6eeb49155f8b9e43a901d5ee22f05275c5d6521d500d34d07b3378c2b: Status 404 returned error can't find the container with id 575036b6eeb49155f8b9e43a901d5ee22f05275c5d6521d500d34d07b3378c2b
Oct 30 19:33:13.088205 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:13.088169    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9" event={"ID":"cd12458e-1fe1-43cf-af8d-cd6c43473c7e","Type":"ContainerStarted","Data":"e1a20d5da980daac8d0529618b5dd35317602837e307298c881068c5a9ce873f"}
Oct 30 19:33:13.088594 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:13.088212    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9" event={"ID":"cd12458e-1fe1-43cf-af8d-cd6c43473c7e","Type":"ContainerStarted","Data":"575036b6eeb49155f8b9e43a901d5ee22f05275c5d6521d500d34d07b3378c2b"}
Oct 30 19:33:13.178375 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:13.178346    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"
Oct 30 19:33:13.200905 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:13.200860    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9" podStartSLOduration=2.200847971 podStartE2EDuration="2.200847971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:13.198926457 +0000 UTC m=+3394.475976901" watchObservedRunningTime="2026-10-30 19:33:13.200847971 +0000 UTC m=+3394.477898416"
Oct 30 19:33:15.710910 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:33:15.710876    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:33:22.178227 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:22.178190    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"
Oct 30 19:33:22.190944 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:22.190921    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"
Oct 30 19:33:23.150144 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:23.150110    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"
Oct 30 19:33:42.038655 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:42.038622    4924 scope.go:122] "RemoveContainer" containerID="3f88422b36ca85663c1da8a6062e11e0799615b36a73050cc83edfdf31c16314"
Oct 30 19:33:42.511770 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:42.511740    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43f207c7-8cfdh"]
Oct 30 19:33:42.516664 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:42.516643    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f207c7-8cfdh"
Oct 30 19:33:42.613737 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:42.613699    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:33:42.613737 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:42.613719    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:33:42.613978 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:42.613796    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:33:42.613978 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:42.613836    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:33:42.615611 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:42.615586    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43f207c7-8cfdh"]
Oct 30 19:33:42.655412 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:42.655370    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a51b9b-5b2e-489d-903f-ea4ace4225b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43f207c7-8cfdh\" (UID: \"45a51b9b-5b2e-489d-903f-ea4ace4225b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f207c7-8cfdh"
Oct 30 19:33:42.655608 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:42.655490    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htvph\" (UniqueName: \"kubernetes.io/projected/45a51b9b-5b2e-489d-903f-ea4ace4225b4-kube-api-access-htvph\") pod \"maas-api-revoke-sub-43f207c7-8cfdh\" (UID: \"45a51b9b-5b2e-489d-903f-ea4ace4225b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f207c7-8cfdh"
Oct 30 19:33:42.655608 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:42.655584    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a51b9b-5b2e-489d-903f-ea4ace4225b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43f207c7-8cfdh\" (UID: \"45a51b9b-5b2e-489d-903f-ea4ace4225b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f207c7-8cfdh"
Oct 30 19:33:42.756510 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:42.756469    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a51b9b-5b2e-489d-903f-ea4ace4225b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43f207c7-8cfdh\" (UID: \"45a51b9b-5b2e-489d-903f-ea4ace4225b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f207c7-8cfdh"
Oct 30 19:33:42.756722 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:42.756561    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htvph\" (UniqueName: \"kubernetes.io/projected/45a51b9b-5b2e-489d-903f-ea4ace4225b4-kube-api-access-htvph\") pod \"maas-api-revoke-sub-43f207c7-8cfdh\" (UID: \"45a51b9b-5b2e-489d-903f-ea4ace4225b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f207c7-8cfdh"
Oct 30 19:33:42.756722 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:42.756664    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a51b9b-5b2e-489d-903f-ea4ace4225b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43f207c7-8cfdh\" (UID: \"45a51b9b-5b2e-489d-903f-ea4ace4225b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f207c7-8cfdh"
Oct 30 19:33:42.757251 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:42.757225    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a51b9b-5b2e-489d-903f-ea4ace4225b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43f207c7-8cfdh\" (UID: \"45a51b9b-5b2e-489d-903f-ea4ace4225b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f207c7-8cfdh"
Oct 30 19:33:42.759016 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:42.758994    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a51b9b-5b2e-489d-903f-ea4ace4225b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43f207c7-8cfdh\" (UID: \"45a51b9b-5b2e-489d-903f-ea4ace4225b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f207c7-8cfdh"
Oct 30 19:33:42.811827 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:42.811752    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htvph\" (UniqueName: \"kubernetes.io/projected/45a51b9b-5b2e-489d-903f-ea4ace4225b4-kube-api-access-htvph\") pod \"maas-api-revoke-sub-43f207c7-8cfdh\" (UID: \"45a51b9b-5b2e-489d-903f-ea4ace4225b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f207c7-8cfdh"
Oct 30 19:33:42.829298 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:42.829270    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f207c7-8cfdh"
Oct 30 19:33:43.198208 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:43.198184    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43f207c7-8cfdh"]
Oct 30 19:33:43.200303 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:33:43.200273    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a51b9b_5b2e_489d_903f_ea4ace4225b4.slice/crio-9ebff03b8f8da51a7ae7c261059d3abfaecbc7878cea3ee9914cf64dfbdb522a WatchSource:0}: Error finding container 9ebff03b8f8da51a7ae7c261059d3abfaecbc7878cea3ee9914cf64dfbdb522a: Status 404 returned error can't find the container with id 9ebff03b8f8da51a7ae7c261059d3abfaecbc7878cea3ee9914cf64dfbdb522a
Oct 30 19:33:43.242027 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:43.241999    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f207c7-8cfdh" event={"ID":"45a51b9b-5b2e-489d-903f-ea4ace4225b4","Type":"ContainerStarted","Data":"9ebff03b8f8da51a7ae7c261059d3abfaecbc7878cea3ee9914cf64dfbdb522a"}
Oct 30 19:33:44.248387 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:44.248349    4924 generic.go:356] "Generic (PLEG): container finished" podID="45a51b9b-5b2e-489d-903f-ea4ace4225b4" containerID="5874a72a5400d50592ea49f138256d45e13baa39827c2715652f4b0e92140759" exitCode=0
Oct 30 19:33:44.248794 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:44.248441    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f207c7-8cfdh" event={"ID":"45a51b9b-5b2e-489d-903f-ea4ace4225b4","Type":"ContainerDied","Data":"5874a72a5400d50592ea49f138256d45e13baa39827c2715652f4b0e92140759"}
Oct 30 19:33:45.392155 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:45.392124    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f207c7-8cfdh"
Oct 30 19:33:45.584240 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:45.584134    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45a51b9b-5b2e-489d-903f-ea4ace4225b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a51b9b-5b2e-489d-903f-ea4ace4225b4-maas-api-openshift-ca\") pod \"45a51b9b-5b2e-489d-903f-ea4ace4225b4\" (UID: \"45a51b9b-5b2e-489d-903f-ea4ace4225b4\") "
Oct 30 19:33:45.584416 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:45.584336    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45a51b9b-5b2e-489d-903f-ea4ace4225b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a51b9b-5b2e-489d-903f-ea4ace4225b4-maas-api-xks-ca\") pod \"45a51b9b-5b2e-489d-903f-ea4ace4225b4\" (UID: \"45a51b9b-5b2e-489d-903f-ea4ace4225b4\") "
Oct 30 19:33:45.584416 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:45.584382    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a51b9b-5b2e-489d-903f-ea4ace4225b4-kube-api-access-htvph\" (UniqueName: \"kubernetes.io/projected/45a51b9b-5b2e-489d-903f-ea4ace4225b4-kube-api-access-htvph\") pod \"45a51b9b-5b2e-489d-903f-ea4ace4225b4\" (UID: \"45a51b9b-5b2e-489d-903f-ea4ace4225b4\") "
Oct 30 19:33:45.584543 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:45.584491    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a51b9b-5b2e-489d-903f-ea4ace4225b4-maas-api-openshift-ca" pod "45a51b9b-5b2e-489d-903f-ea4ace4225b4" (UID: "45a51b9b-5b2e-489d-903f-ea4ace4225b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:33:45.584708 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:45.584692    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45a51b9b-5b2e-489d-903f-ea4ace4225b4-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:45.586539 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:45.586508    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a51b9b-5b2e-489d-903f-ea4ace4225b4-kube-api-access-htvph" pod "45a51b9b-5b2e-489d-903f-ea4ace4225b4" (UID: "45a51b9b-5b2e-489d-903f-ea4ace4225b4"). InnerVolumeSpecName "kube-api-access-htvph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:33:45.586679 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:45.586553    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45a51b9b-5b2e-489d-903f-ea4ace4225b4-maas-api-xks-ca" pod "45a51b9b-5b2e-489d-903f-ea4ace4225b4" (UID: "45a51b9b-5b2e-489d-903f-ea4ace4225b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:33:45.685428 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:45.685393    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45a51b9b-5b2e-489d-903f-ea4ace4225b4-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:45.685428 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:45.685419    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htvph\" (UniqueName: \"kubernetes.io/projected/45a51b9b-5b2e-489d-903f-ea4ace4225b4-kube-api-access-htvph\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:45.711154 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:33:45.711113    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:33:46.260060 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:46.260024    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f207c7-8cfdh" event={"ID":"45a51b9b-5b2e-489d-903f-ea4ace4225b4","Type":"ContainerDied","Data":"9ebff03b8f8da51a7ae7c261059d3abfaecbc7878cea3ee9914cf64dfbdb522a"}
Oct 30 19:33:46.260060 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:46.260051    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f207c7-8cfdh"
Oct 30 19:33:46.260313 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:46.260097    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ebff03b8f8da51a7ae7c261059d3abfaecbc7878cea3ee9914cf64dfbdb522a"
Oct 30 19:33:53.895097 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:53.895053    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl"]
Oct 30 19:33:53.895565 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:53.895450    4924 cpu_manager.go:414] "containerMap: removing container" podUID="45a51b9b-5b2e-489d-903f-ea4ace4225b4" containerName="revoke-keys"
Oct 30 19:33:53.895565 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:53.895462    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a51b9b-5b2e-489d-903f-ea4ace4225b4" containerName="revoke-keys"
Oct 30 19:33:53.895565 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:53.895549    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="45a51b9b-5b2e-489d-903f-ea4ace4225b4" containerName="revoke-keys"
Oct 30 19:33:53.899293 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:53.899272    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl"
Oct 30 19:33:53.954293 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:53.954262    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:33:53.960754 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:53.960721    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df49757-e39f-41e8-a264-61c4da5b21ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl\" (UID: \"1df49757-e39f-41e8-a264-61c4da5b21ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl"
Oct 30 19:33:53.960906 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:53.960760    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjfct\" (UniqueName: \"kubernetes.io/projected/1df49757-e39f-41e8-a264-61c4da5b21ae-kube-api-access-gjfct\") pod \"maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl\" (UID: \"1df49757-e39f-41e8-a264-61c4da5b21ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl"
Oct 30 19:33:53.960906 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:53.960850    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df49757-e39f-41e8-a264-61c4da5b21ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl\" (UID: \"1df49757-e39f-41e8-a264-61c4da5b21ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl"
Oct 30 19:33:54.000807 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:54.000778    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:33:54.000991 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:54.000891    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:33:54.000991 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:54.000928    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:33:54.003016 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:54.002991    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl"]
Oct 30 19:33:54.062182 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:54.062143    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df49757-e39f-41e8-a264-61c4da5b21ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl\" (UID: \"1df49757-e39f-41e8-a264-61c4da5b21ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl"
Oct 30 19:33:54.062182 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:54.062189    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjfct\" (UniqueName: \"kubernetes.io/projected/1df49757-e39f-41e8-a264-61c4da5b21ae-kube-api-access-gjfct\") pod \"maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl\" (UID: \"1df49757-e39f-41e8-a264-61c4da5b21ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl"
Oct 30 19:33:54.062413 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:54.062227    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df49757-e39f-41e8-a264-61c4da5b21ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl\" (UID: \"1df49757-e39f-41e8-a264-61c4da5b21ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl"
Oct 30 19:33:54.062907 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:54.062883    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df49757-e39f-41e8-a264-61c4da5b21ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl\" (UID: \"1df49757-e39f-41e8-a264-61c4da5b21ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl"
Oct 30 19:33:54.064661 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:54.064635    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df49757-e39f-41e8-a264-61c4da5b21ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl\" (UID: \"1df49757-e39f-41e8-a264-61c4da5b21ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl"
Oct 30 19:33:54.120539 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:54.120502    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjfct\" (UniqueName: \"kubernetes.io/projected/1df49757-e39f-41e8-a264-61c4da5b21ae-kube-api-access-gjfct\") pod \"maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl\" (UID: \"1df49757-e39f-41e8-a264-61c4da5b21ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl"
Oct 30 19:33:54.211214 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:54.211129    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl"
Oct 30 19:33:54.594670 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:54.594632    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl"]
Oct 30 19:33:54.595973 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:33:54.595947    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df49757_e39f_41e8_a264_61c4da5b21ae.slice/crio-e6f1f5d6ba6514a63e07c22c85cad6744b4f9a63f441ab179917e72a69698ca5 WatchSource:0}: Error finding container e6f1f5d6ba6514a63e07c22c85cad6744b4f9a63f441ab179917e72a69698ca5: Status 404 returned error can't find the container with id e6f1f5d6ba6514a63e07c22c85cad6744b4f9a63f441ab179917e72a69698ca5
Oct 30 19:33:55.308868 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:55.308770    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl" event={"ID":"1df49757-e39f-41e8-a264-61c4da5b21ae","Type":"ContainerStarted","Data":"2582545d37d25ae9d2e34614cf24f8e803b7ff64e7ddda007a8f435c41f96129"}
Oct 30 19:33:55.308868 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:55.308832    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl" event={"ID":"1df49757-e39f-41e8-a264-61c4da5b21ae","Type":"ContainerStarted","Data":"e6f1f5d6ba6514a63e07c22c85cad6744b4f9a63f441ab179917e72a69698ca5"}
Oct 30 19:33:55.420848 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:55.420788    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl" podStartSLOduration=2.420773246 podStartE2EDuration="2.420773246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:55.418648616 +0000 UTC m=+3436.695699061" watchObservedRunningTime="2026-10-30 19:33:55.420773246 +0000 UTC m=+3436.697823687"
Oct 30 19:33:56.315351 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:56.315316    4924 generic.go:356] "Generic (PLEG): container finished" podID="1df49757-e39f-41e8-a264-61c4da5b21ae" containerID="2582545d37d25ae9d2e34614cf24f8e803b7ff64e7ddda007a8f435c41f96129" exitCode=0
Oct 30 19:33:56.315723 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:56.315399    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl" event={"ID":"1df49757-e39f-41e8-a264-61c4da5b21ae","Type":"ContainerDied","Data":"2582545d37d25ae9d2e34614cf24f8e803b7ff64e7ddda007a8f435c41f96129"}
Oct 30 19:33:57.451543 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:57.451518    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl"
Oct 30 19:33:57.498697 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:57.498658    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1df49757-e39f-41e8-a264-61c4da5b21ae-kube-api-access-gjfct\" (UniqueName: \"kubernetes.io/projected/1df49757-e39f-41e8-a264-61c4da5b21ae-kube-api-access-gjfct\") pod \"1df49757-e39f-41e8-a264-61c4da5b21ae\" (UID: \"1df49757-e39f-41e8-a264-61c4da5b21ae\") "
Oct 30 19:33:57.498872 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:57.498753    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1df49757-e39f-41e8-a264-61c4da5b21ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df49757-e39f-41e8-a264-61c4da5b21ae-maas-api-openshift-ca\") pod \"1df49757-e39f-41e8-a264-61c4da5b21ae\" (UID: \"1df49757-e39f-41e8-a264-61c4da5b21ae\") "
Oct 30 19:33:57.498872 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:57.498839    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1df49757-e39f-41e8-a264-61c4da5b21ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df49757-e39f-41e8-a264-61c4da5b21ae-maas-api-xks-ca\") pod \"1df49757-e39f-41e8-a264-61c4da5b21ae\" (UID: \"1df49757-e39f-41e8-a264-61c4da5b21ae\") "
Oct 30 19:33:57.499199 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:57.499162    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1df49757-e39f-41e8-a264-61c4da5b21ae-maas-api-openshift-ca" pod "1df49757-e39f-41e8-a264-61c4da5b21ae" (UID: "1df49757-e39f-41e8-a264-61c4da5b21ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:33:57.501028 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:57.500998    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1df49757-e39f-41e8-a264-61c4da5b21ae-maas-api-xks-ca" pod "1df49757-e39f-41e8-a264-61c4da5b21ae" (UID: "1df49757-e39f-41e8-a264-61c4da5b21ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:33:57.501167 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:57.501083    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1df49757-e39f-41e8-a264-61c4da5b21ae-kube-api-access-gjfct" pod "1df49757-e39f-41e8-a264-61c4da5b21ae" (UID: "1df49757-e39f-41e8-a264-61c4da5b21ae"). InnerVolumeSpecName "kube-api-access-gjfct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:33:57.599825 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:57.599791    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1df49757-e39f-41e8-a264-61c4da5b21ae-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:57.599825 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:57.599824    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjfct\" (UniqueName: \"kubernetes.io/projected/1df49757-e39f-41e8-a264-61c4da5b21ae-kube-api-access-gjfct\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:57.600036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:57.599835    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1df49757-e39f-41e8-a264-61c4da5b21ae-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:58.327593 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:58.327561    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl"
Oct 30 19:33:58.327804 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:58.327561    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl" event={"ID":"1df49757-e39f-41e8-a264-61c4da5b21ae","Type":"ContainerDied","Data":"e6f1f5d6ba6514a63e07c22c85cad6744b4f9a63f441ab179917e72a69698ca5"}
Oct 30 19:33:58.327804 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:58.327696    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6f1f5d6ba6514a63e07c22c85cad6744b4f9a63f441ab179917e72a69698ca5"
Oct 30 19:33:58.561009 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:33:58.560972    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"]
Oct 30 19:34:05.254737 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:05.254696    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk"]
Oct 30 19:34:10.255816 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:10.255745    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk" podUID="135a7159-612b-432b-9010-dcc1a0a0e071" containerName="payload-processing" containerID="cri-o://e2d8d23aa59dc47ea219bc3d5b909c167d4396b509d2ca6bde5f7da75348722a" gracePeriod=25
Oct 30 19:34:11.285849 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:11.285810    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl"]
Oct 30 19:34:11.332606 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:11.332571    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-34d64c-bp7tl"]
Oct 30 19:34:12.833373 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:12.833341    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1df49757-e39f-41e8-a264-61c4da5b21ae" path="/var/lib/kubelet/pods/1df49757-e39f-41e8-a264-61c4da5b21ae/volumes"
Oct 30 19:34:13.581226 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:13.581160    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9" podUID="cd12458e-1fe1-43cf-af8d-cd6c43473c7e" containerName="main" containerID="cri-o://e1a20d5da980daac8d0529618b5dd35317602837e307298c881068c5a9ce873f" gracePeriod=45
Oct 30 19:34:13.839035 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:13.838970    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"
Oct 30 19:34:13.961899 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:13.961862    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-kube-api-access-ftm9q\" (UniqueName: \"kubernetes.io/projected/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-kube-api-access-ftm9q\") pod \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\" (UID: \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\") "
Oct 30 19:34:13.962118 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:13.961979    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-tls-certs\") pod \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\" (UID: \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\") "
Oct 30 19:34:13.962118 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:13.962034    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-dshm\") pod \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\" (UID: \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\") "
Oct 30 19:34:13.962285 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:13.962161    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-model-cache\") pod \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\" (UID: \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\") "
Oct 30 19:34:13.962285 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:13.962200    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-tmp-dir\") pod \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\" (UID: \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\") "
Oct 30 19:34:13.962285 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:13.962242    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-home\" (UniqueName: \"kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-home\") pod \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\" (UID: \"cd12458e-1fe1-43cf-af8d-cd6c43473c7e\") "
Oct 30 19:34:13.962484 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:13.962456    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-model-cache" pod "cd12458e-1fe1-43cf-af8d-cd6c43473c7e" (UID: "cd12458e-1fe1-43cf-af8d-cd6c43473c7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:13.962757 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:13.962735    4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-model-cache\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:13.962895 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:13.962869    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-tmp-dir" pod "cd12458e-1fe1-43cf-af8d-cd6c43473c7e" (UID: "cd12458e-1fe1-43cf-af8d-cd6c43473c7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:13.962967 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:13.962943    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-home" pod "cd12458e-1fe1-43cf-af8d-cd6c43473c7e" (UID: "cd12458e-1fe1-43cf-af8d-cd6c43473c7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:13.964405 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:13.964380    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-kube-api-access-ftm9q" pod "cd12458e-1fe1-43cf-af8d-cd6c43473c7e" (UID: "cd12458e-1fe1-43cf-af8d-cd6c43473c7e"). InnerVolumeSpecName "kube-api-access-ftm9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:13.964596 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:13.964571    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-dshm" pod "cd12458e-1fe1-43cf-af8d-cd6c43473c7e" (UID: "cd12458e-1fe1-43cf-af8d-cd6c43473c7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:13.964596 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:13.964581    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-tls-certs" pod "cd12458e-1fe1-43cf-af8d-cd6c43473c7e" (UID: "cd12458e-1fe1-43cf-af8d-cd6c43473c7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:34:14.063716 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:14.063679    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftm9q\" (UniqueName: \"kubernetes.io/projected/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-kube-api-access-ftm9q\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:14.063716 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:14.063716    4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-tls-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:14.063934 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:14.063730    4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-dshm\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:14.063934 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:14.063741    4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-tmp-dir\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:14.063934 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:14.063752    4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd12458e-1fe1-43cf-af8d-cd6c43473c7e-home\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:14.410456 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:14.410420    4924 generic.go:356] "Generic (PLEG): container finished" podID="cd12458e-1fe1-43cf-af8d-cd6c43473c7e" containerID="e1a20d5da980daac8d0529618b5dd35317602837e307298c881068c5a9ce873f" exitCode=0
Oct 30 19:34:14.410670 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:14.410465    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9" event={"ID":"cd12458e-1fe1-43cf-af8d-cd6c43473c7e","Type":"ContainerDied","Data":"e1a20d5da980daac8d0529618b5dd35317602837e307298c881068c5a9ce873f"}
Oct 30 19:34:14.410670 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:14.410475    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"
Oct 30 19:34:14.410670 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:14.410499    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9" event={"ID":"cd12458e-1fe1-43cf-af8d-cd6c43473c7e","Type":"ContainerDied","Data":"575036b6eeb49155f8b9e43a901d5ee22f05275c5d6521d500d34d07b3378c2b"}
Oct 30 19:34:14.410670 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:14.410516    4924 scope.go:122] "RemoveContainer" containerID="e1a20d5da980daac8d0529618b5dd35317602837e307298c881068c5a9ce873f"
Oct 30 19:34:14.420306 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:14.420290    4924 scope.go:122] "RemoveContainer" containerID="e1a20d5da980daac8d0529618b5dd35317602837e307298c881068c5a9ce873f"
Oct 30 19:34:14.420571 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:34:14.420549    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1a20d5da980daac8d0529618b5dd35317602837e307298c881068c5a9ce873f\": container with ID starting with e1a20d5da980daac8d0529618b5dd35317602837e307298c881068c5a9ce873f not found: ID does not exist" containerID="e1a20d5da980daac8d0529618b5dd35317602837e307298c881068c5a9ce873f"
Oct 30 19:34:14.420672 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:14.420577    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1a20d5da980daac8d0529618b5dd35317602837e307298c881068c5a9ce873f"} err="failed to get container status \"e1a20d5da980daac8d0529618b5dd35317602837e307298c881068c5a9ce873f\": rpc error: code = NotFound desc = could not find container \"e1a20d5da980daac8d0529618b5dd35317602837e307298c881068c5a9ce873f\": container with ID starting with e1a20d5da980daac8d0529618b5dd35317602837e307298c881068c5a9ce873f not found: ID does not exist"
Oct 30 19:34:14.567184 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:14.567146    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"]
Oct 30 19:34:14.613648 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:14.613615    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-34d64c/simulator-single-main-34d64c-kserve-55b74b76fd-676j9"]
Oct 30 19:34:14.834749 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:14.834660    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd12458e-1fe1-43cf-af8d-cd6c43473c7e" path="/var/lib/kubelet/pods/cd12458e-1fe1-43cf-af8d-cd6c43473c7e/volumes"
Oct 30 19:34:15.710491 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:34:15.710453    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:34:19.975842 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:19.975758    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk" podUID="135a7159-612b-432b-9010-dcc1a0a0e071" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:34:25.054363 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:25.054328    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7de1cfd4-7vghp"]
Oct 30 19:34:25.100697 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:25.100665    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7de1cfd4-7vghp"]
Oct 30 19:34:26.833456 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:26.833420    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d563938-c50f-47b4-b74f-1ae0648085fd" path="/var/lib/kubelet/pods/2d563938-c50f-47b4-b74f-1ae0648085fd/volumes"
Oct 30 19:34:29.976717 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:29.976673    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk" podUID="135a7159-612b-432b-9010-dcc1a0a0e071" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:34:30.499556 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:30.499367    4924 generic.go:356] "Generic (PLEG): container finished" podID="135a7159-612b-432b-9010-dcc1a0a0e071" containerID="e2d8d23aa59dc47ea219bc3d5b909c167d4396b509d2ca6bde5f7da75348722a" exitCode=0
Oct 30 19:34:30.499556 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:30.499481    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk" event={"ID":"135a7159-612b-432b-9010-dcc1a0a0e071","Type":"ContainerDied","Data":"e2d8d23aa59dc47ea219bc3d5b909c167d4396b509d2ca6bde5f7da75348722a"}
Oct 30 19:34:31.515312 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:31.515279    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk"
Oct 30 19:34:31.529227 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:31.529188    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk" event={"ID":"135a7159-612b-432b-9010-dcc1a0a0e071","Type":"ContainerDied","Data":"15af697c5869270510f24842dfd9dd927a55b971bd094045e51ce5a8f0c0af7d"}
Oct 30 19:34:31.531665 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:31.531641    4924 scope.go:122] "RemoveContainer" containerID="e2d8d23aa59dc47ea219bc3d5b909c167d4396b509d2ca6bde5f7da75348722a"
Oct 30 19:34:31.712090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:31.692948    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/135a7159-612b-432b-9010-dcc1a0a0e071-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/135a7159-612b-432b-9010-dcc1a0a0e071-plugins-config-volume\") pod \"135a7159-612b-432b-9010-dcc1a0a0e071\" (UID: \"135a7159-612b-432b-9010-dcc1a0a0e071\") "
Oct 30 19:34:31.712090 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:31.693002    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135a7159-612b-432b-9010-dcc1a0a0e071-kube-api-access-jt987\" (UniqueName: \"kubernetes.io/projected/135a7159-612b-432b-9010-dcc1a0a0e071-kube-api-access-jt987\") pod \"135a7159-612b-432b-9010-dcc1a0a0e071\" (UID: \"135a7159-612b-432b-9010-dcc1a0a0e071\") "
Oct 30 19:34:31.712825 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:31.712757    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135a7159-612b-432b-9010-dcc1a0a0e071-plugins-config-volume" pod "135a7159-612b-432b-9010-dcc1a0a0e071" (UID: "135a7159-612b-432b-9010-dcc1a0a0e071"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:31.774312 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:31.774273    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135a7159-612b-432b-9010-dcc1a0a0e071-kube-api-access-jt987" pod "135a7159-612b-432b-9010-dcc1a0a0e071" (UID: "135a7159-612b-432b-9010-dcc1a0a0e071"). InnerVolumeSpecName "kube-api-access-jt987". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:31.796080 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:31.795845    4924 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/135a7159-612b-432b-9010-dcc1a0a0e071-plugins-config-volume\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:31.796080 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:31.795887    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt987\" (UniqueName: \"kubernetes.io/projected/135a7159-612b-432b-9010-dcc1a0a0e071-kube-api-access-jt987\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:32.258592 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:32.258168    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-4kcpg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:34:32.258592 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:32.258247    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg" podUID="db7e180c-7647-4b96-be77-c7d913ae4270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:32.480227 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:32.480181    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jvm6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:34:32.480602 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:32.480566    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jvm6x" podUID="75c996cd-4196-4e8a-b067-9fc146104887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:34:32.542329 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:32.542297    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk"
Oct 30 19:34:32.703121 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:32.703087    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk"]
Oct 30 19:34:32.705515 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:32.705484    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-34d64c-78fd467c6c7dwk"]
Oct 30 19:34:32.833056 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:32.832972    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="135a7159-612b-432b-9010-dcc1a0a0e071" path="/var/lib/kubelet/pods/135a7159-612b-432b-9010-dcc1a0a0e071/volumes"
Oct 30 19:34:42.058624 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:42.058593    4924 scope.go:122] "RemoveContainer" containerID="699c5e5b756aaf1ab81cb39078f8ef856229b0e3b7ffed12303d290d33e1002a"
Oct 30 19:34:45.710593 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:34:45.710558    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:34:47.846370 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.846336    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"]
Oct 30 19:34:47.846964 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.846893    4924 cpu_manager.go:414] "containerMap: removing container" podUID="cd12458e-1fe1-43cf-af8d-cd6c43473c7e" containerName="main"
Oct 30 19:34:47.846964 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.846914    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd12458e-1fe1-43cf-af8d-cd6c43473c7e" containerName="main"
Oct 30 19:34:47.846964 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.846954    4924 cpu_manager.go:414] "containerMap: removing container" podUID="135a7159-612b-432b-9010-dcc1a0a0e071" containerName="payload-processing"
Oct 30 19:34:47.846964 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.846962    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135a7159-612b-432b-9010-dcc1a0a0e071" containerName="payload-processing"
Oct 30 19:34:47.847225 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.846977    4924 cpu_manager.go:414] "containerMap: removing container" podUID="1df49757-e39f-41e8-a264-61c4da5b21ae" containerName="revoke-keys"
Oct 30 19:34:47.847225 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.846985    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1df49757-e39f-41e8-a264-61c4da5b21ae" containerName="revoke-keys"
Oct 30 19:34:47.847225 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.847124    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd12458e-1fe1-43cf-af8d-cd6c43473c7e" containerName="main"
Oct 30 19:34:47.847225 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.847143    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="135a7159-612b-432b-9010-dcc1a0a0e071" containerName="payload-processing"
Oct 30 19:34:47.847225 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.847155    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="1df49757-e39f-41e8-a264-61c4da5b21ae" containerName="revoke-keys"
Oct 30 19:34:47.851582 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.851563    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"
Oct 30 19:34:47.887559 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.887512    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/026b280e-c549-4143-a61d-37e4d48f97b3-tls-certs\") pod \"e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v\" (UID: \"026b280e-c549-4143-a61d-37e4d48f97b3\") " pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"
Oct 30 19:34:47.887761 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.887584    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-model-cache\") pod \"e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v\" (UID: \"026b280e-c549-4143-a61d-37e4d48f97b3\") " pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"
Oct 30 19:34:47.887761 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.887607    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n95r7\" (UniqueName: \"kubernetes.io/projected/026b280e-c549-4143-a61d-37e4d48f97b3-kube-api-access-n95r7\") pod \"e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v\" (UID: \"026b280e-c549-4143-a61d-37e4d48f97b3\") " pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"
Oct 30 19:34:47.887892 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.887849    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-dshm\") pod \"e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v\" (UID: \"026b280e-c549-4143-a61d-37e4d48f97b3\") " pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"
Oct 30 19:34:47.887954 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.887903    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-tmp-dir\") pod \"e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v\" (UID: \"026b280e-c549-4143-a61d-37e4d48f97b3\") " pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"
Oct 30 19:34:47.888024 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.887967    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-home\") pod \"e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v\" (UID: \"026b280e-c549-4143-a61d-37e4d48f97b3\") " pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"
Oct 30 19:34:47.950728 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.950692    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"]
Oct 30 19:34:47.956997 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.956974    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-6a305904-kserve-self-signed-certs\""
Oct 30 19:34:47.989252 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.989214    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/026b280e-c549-4143-a61d-37e4d48f97b3-tls-certs\") pod \"e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v\" (UID: \"026b280e-c549-4143-a61d-37e4d48f97b3\") " pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"
Oct 30 19:34:47.989418 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.989315    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-model-cache\") pod \"e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v\" (UID: \"026b280e-c549-4143-a61d-37e4d48f97b3\") " pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"
Oct 30 19:34:47.989418 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.989347    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n95r7\" (UniqueName: \"kubernetes.io/projected/026b280e-c549-4143-a61d-37e4d48f97b3-kube-api-access-n95r7\") pod \"e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v\" (UID: \"026b280e-c549-4143-a61d-37e4d48f97b3\") " pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"
Oct 30 19:34:47.989418 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.989383    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-dshm\") pod \"e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v\" (UID: \"026b280e-c549-4143-a61d-37e4d48f97b3\") " pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"
Oct 30 19:34:47.989610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.989431    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-tmp-dir\") pod \"e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v\" (UID: \"026b280e-c549-4143-a61d-37e4d48f97b3\") " pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"
Oct 30 19:34:47.989610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.989493    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-home\") pod \"e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v\" (UID: \"026b280e-c549-4143-a61d-37e4d48f97b3\") " pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"
Oct 30 19:34:47.990033 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.990006    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-tmp-dir\") pod \"e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v\" (UID: \"026b280e-c549-4143-a61d-37e4d48f97b3\") " pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"
Oct 30 19:34:47.990196 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.990132    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-home\") pod \"e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v\" (UID: \"026b280e-c549-4143-a61d-37e4d48f97b3\") " pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"
Oct 30 19:34:47.990196 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.990156    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-model-cache\") pod \"e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v\" (UID: \"026b280e-c549-4143-a61d-37e4d48f97b3\") " pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"
Oct 30 19:34:47.993012 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.992984    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/026b280e-c549-4143-a61d-37e4d48f97b3-tls-certs\") pod \"e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v\" (UID: \"026b280e-c549-4143-a61d-37e4d48f97b3\") " pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"
Oct 30 19:34:47.993153 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:47.993135    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-dshm\") pod \"e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v\" (UID: \"026b280e-c549-4143-a61d-37e4d48f97b3\") " pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"
Oct 30 19:34:48.044139 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:48.044100    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n95r7\" (UniqueName: \"kubernetes.io/projected/026b280e-c549-4143-a61d-37e4d48f97b3-kube-api-access-n95r7\") pod \"e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v\" (UID: \"026b280e-c549-4143-a61d-37e4d48f97b3\") " pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"
Oct 30 19:34:48.163343 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:48.163256    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"
Oct 30 19:34:48.537097 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:48.537059    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"]
Oct 30 19:34:48.539232 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:34:48.539207    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod026b280e_c549_4143_a61d_37e4d48f97b3.slice/crio-854ee9085bc73e9a83bcce3bb2abd1c922fd962575e32a23370119db2bfebda1 WatchSource:0}: Error finding container 854ee9085bc73e9a83bcce3bb2abd1c922fd962575e32a23370119db2bfebda1: Status 404 returned error can't find the container with id 854ee9085bc73e9a83bcce3bb2abd1c922fd962575e32a23370119db2bfebda1
Oct 30 19:34:48.630127 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:48.630093    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v" event={"ID":"026b280e-c549-4143-a61d-37e4d48f97b3","Type":"ContainerStarted","Data":"f974c7ebfcbf892f37313b7668e0fedad73eaea3d98f2b412995b12d1a2827b5"}
Oct 30 19:34:48.630311 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:48.630139    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v" event={"ID":"026b280e-c549-4143-a61d-37e4d48f97b3","Type":"ContainerStarted","Data":"854ee9085bc73e9a83bcce3bb2abd1c922fd962575e32a23370119db2bfebda1"}
Oct 30 19:34:48.741845 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:48.741786    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v" podStartSLOduration=1.741767796 podStartE2EDuration="1.741767796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:48.740879607 +0000 UTC m=+3490.017930067" watchObservedRunningTime="2026-10-30 19:34:48.741767796 +0000 UTC m=+3490.018818241"
Oct 30 19:34:49.163758 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:49.163717    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"
Oct 30 19:34:58.163649 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:58.163602    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"
Oct 30 19:34:58.176458 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:58.176434    4924 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"
Oct 30 19:34:58.727213 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:34:58.727181    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"
Oct 30 19:35:15.712130 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:35:15.712093    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:35:16.095970 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:16.095897    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf6xw"]
Oct 30 19:35:16.099291 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:16.099272    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf6xw"
Oct 30 19:35:16.148910 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:16.148880    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:35:16.148910 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:16.148902    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:35:16.195472 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:16.195445    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:35:16.195739 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:16.195721    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:35:16.197610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:16.197589    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf6xw"]
Oct 30 19:35:16.249943 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:16.249918    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2pbf\" (UniqueName: \"kubernetes.io/projected/2185cb6f-0366-4545-9a0a-72ea26d6cadf-kube-api-access-f2pbf\") pod \"maas-api-revoke-sub-452b6d7b-gf6xw\" (UID: \"2185cb6f-0366-4545-9a0a-72ea26d6cadf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf6xw"
Oct 30 19:35:16.250124 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:16.249950    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2185cb6f-0366-4545-9a0a-72ea26d6cadf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-gf6xw\" (UID: \"2185cb6f-0366-4545-9a0a-72ea26d6cadf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf6xw"
Oct 30 19:35:16.250124 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:16.250111    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2185cb6f-0366-4545-9a0a-72ea26d6cadf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-gf6xw\" (UID: \"2185cb6f-0366-4545-9a0a-72ea26d6cadf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf6xw"
Oct 30 19:35:16.351000 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:16.350972    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2pbf\" (UniqueName: \"kubernetes.io/projected/2185cb6f-0366-4545-9a0a-72ea26d6cadf-kube-api-access-f2pbf\") pod \"maas-api-revoke-sub-452b6d7b-gf6xw\" (UID: \"2185cb6f-0366-4545-9a0a-72ea26d6cadf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf6xw"
Oct 30 19:35:16.351197 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:16.351009    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2185cb6f-0366-4545-9a0a-72ea26d6cadf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-gf6xw\" (UID: \"2185cb6f-0366-4545-9a0a-72ea26d6cadf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf6xw"
Oct 30 19:35:16.351197 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:16.351106    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2185cb6f-0366-4545-9a0a-72ea26d6cadf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-gf6xw\" (UID: \"2185cb6f-0366-4545-9a0a-72ea26d6cadf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf6xw"
Oct 30 19:35:16.351729 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:16.351704    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2185cb6f-0366-4545-9a0a-72ea26d6cadf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-gf6xw\" (UID: \"2185cb6f-0366-4545-9a0a-72ea26d6cadf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf6xw"
Oct 30 19:35:16.353408 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:16.353389    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2185cb6f-0366-4545-9a0a-72ea26d6cadf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-gf6xw\" (UID: \"2185cb6f-0366-4545-9a0a-72ea26d6cadf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf6xw"
Oct 30 19:35:16.405962 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:16.405940    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2pbf\" (UniqueName: \"kubernetes.io/projected/2185cb6f-0366-4545-9a0a-72ea26d6cadf-kube-api-access-f2pbf\") pod \"maas-api-revoke-sub-452b6d7b-gf6xw\" (UID: \"2185cb6f-0366-4545-9a0a-72ea26d6cadf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf6xw"
Oct 30 19:35:16.411776 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:16.411751    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf6xw"
Oct 30 19:35:16.780327 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:16.780298    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf6xw"]
Oct 30 19:35:16.782519 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:35:16.782486    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2185cb6f_0366_4545_9a0a_72ea26d6cadf.slice/crio-9f6a86da1c0e026dd793e291bc9c53d51f6e45f6ea3e235e51f9bfb393d51757 WatchSource:0}: Error finding container 9f6a86da1c0e026dd793e291bc9c53d51f6e45f6ea3e235e51f9bfb393d51757: Status 404 returned error can't find the container with id 9f6a86da1c0e026dd793e291bc9c53d51f6e45f6ea3e235e51f9bfb393d51757
Oct 30 19:35:16.804003 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:16.803978    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf6xw" event={"ID":"2185cb6f-0366-4545-9a0a-72ea26d6cadf","Type":"ContainerStarted","Data":"9f6a86da1c0e026dd793e291bc9c53d51f6e45f6ea3e235e51f9bfb393d51757"}
Oct 30 19:35:17.813714 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:17.813673    4924 generic.go:356] "Generic (PLEG): container finished" podID="2185cb6f-0366-4545-9a0a-72ea26d6cadf" containerID="4c232e12a3c740bdb34c403b05f729d64d7272f5e5283f0099ab09618b6d4bc9" exitCode=0
Oct 30 19:35:17.814254 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:17.813766    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf6xw" event={"ID":"2185cb6f-0366-4545-9a0a-72ea26d6cadf","Type":"ContainerDied","Data":"4c232e12a3c740bdb34c403b05f729d64d7272f5e5283f0099ab09618b6d4bc9"}
Oct 30 19:35:19.782690 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:19.782655    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf6xw"
Oct 30 19:35:19.806413 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:19.806381    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2185cb6f-0366-4545-9a0a-72ea26d6cadf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2185cb6f-0366-4545-9a0a-72ea26d6cadf-maas-api-openshift-ca\") pod \"2185cb6f-0366-4545-9a0a-72ea26d6cadf\" (UID: \"2185cb6f-0366-4545-9a0a-72ea26d6cadf\") "
Oct 30 19:35:19.822550 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:19.822524    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2185cb6f-0366-4545-9a0a-72ea26d6cadf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2185cb6f-0366-4545-9a0a-72ea26d6cadf-maas-api-xks-ca\") pod \"2185cb6f-0366-4545-9a0a-72ea26d6cadf\" (UID: \"2185cb6f-0366-4545-9a0a-72ea26d6cadf\") "
Oct 30 19:35:19.823176 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:19.823154    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2185cb6f-0366-4545-9a0a-72ea26d6cadf-kube-api-access-f2pbf\" (UniqueName: \"kubernetes.io/projected/2185cb6f-0366-4545-9a0a-72ea26d6cadf-kube-api-access-f2pbf\") pod \"2185cb6f-0366-4545-9a0a-72ea26d6cadf\" (UID: \"2185cb6f-0366-4545-9a0a-72ea26d6cadf\") "
Oct 30 19:35:19.823523 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:19.822857    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2185cb6f-0366-4545-9a0a-72ea26d6cadf-maas-api-openshift-ca" pod "2185cb6f-0366-4545-9a0a-72ea26d6cadf" (UID: "2185cb6f-0366-4545-9a0a-72ea26d6cadf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:19.824912 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:19.824891    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2185cb6f-0366-4545-9a0a-72ea26d6cadf-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:19.851755 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:19.845555    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2185cb6f-0366-4545-9a0a-72ea26d6cadf-maas-api-xks-ca" pod "2185cb6f-0366-4545-9a0a-72ea26d6cadf" (UID: "2185cb6f-0366-4545-9a0a-72ea26d6cadf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:35:19.852176 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:19.852135    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2185cb6f-0366-4545-9a0a-72ea26d6cadf-kube-api-access-f2pbf" pod "2185cb6f-0366-4545-9a0a-72ea26d6cadf" (UID: "2185cb6f-0366-4545-9a0a-72ea26d6cadf"). InnerVolumeSpecName "kube-api-access-f2pbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:19.883038 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:19.882964    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf6xw" event={"ID":"2185cb6f-0366-4545-9a0a-72ea26d6cadf","Type":"ContainerDied","Data":"9f6a86da1c0e026dd793e291bc9c53d51f6e45f6ea3e235e51f9bfb393d51757"}
Oct 30 19:35:19.883259 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:19.883241    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f6a86da1c0e026dd793e291bc9c53d51f6e45f6ea3e235e51f9bfb393d51757"
Oct 30 19:35:19.883424 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:19.883412    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf6xw"
Oct 30 19:35:19.933764 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:19.933731    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2185cb6f-0366-4545-9a0a-72ea26d6cadf-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:19.934798 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:19.934775    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2pbf\" (UniqueName: \"kubernetes.io/projected/2185cb6f-0366-4545-9a0a-72ea26d6cadf-kube-api-access-f2pbf\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:38.082576 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:38.082545    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"]
Oct 30 19:35:42.093901 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:42.093863    4924 scope.go:122] "RemoveContainer" containerID="1676385fa104245d0cc13dfff5377397bd9454b664579b32faf3c480bfa4fee0"
Oct 30 19:35:42.102502 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:42.102479    4924 scope.go:122] "RemoveContainer" containerID="cca84ec456dba2c3fecb077fcf3a2ff9ab1af7dc75fe7cbf75e449f14fa01355"
Oct 30 19:35:42.169476 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:42.169447    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gpbc5"]
Oct 30 19:35:42.169868 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:42.169854    4924 cpu_manager.go:414] "containerMap: removing container" podUID="2185cb6f-0366-4545-9a0a-72ea26d6cadf" containerName="revoke-keys"
Oct 30 19:35:42.169868 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:42.169867    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2185cb6f-0366-4545-9a0a-72ea26d6cadf" containerName="revoke-keys"
Oct 30 19:35:42.169985 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:42.169956    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="2185cb6f-0366-4545-9a0a-72ea26d6cadf" containerName="revoke-keys"
Oct 30 19:35:42.172977 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:42.172958    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gpbc5"
Oct 30 19:35:42.225206 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:42.225179    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:35:42.227278 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:42.227248    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gpbc5"]
Oct 30 19:35:42.270932 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:42.270904    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:35:42.270932 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:42.270904    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:35:42.271142 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:42.270966    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:35:42.285852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:42.285829    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ae31749-2655-4595-8089-6972597b80f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-gpbc5\" (UID: \"3ae31749-2655-4595-8089-6972597b80f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gpbc5"
Oct 30 19:35:42.286014 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:42.285928    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8wph\" (UniqueName: \"kubernetes.io/projected/3ae31749-2655-4595-8089-6972597b80f0-kube-api-access-n8wph\") pod \"maas-api-revoke-sub-3e31a506-gpbc5\" (UID: \"3ae31749-2655-4595-8089-6972597b80f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gpbc5"
Oct 30 19:35:42.286096 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:42.286017    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ae31749-2655-4595-8089-6972597b80f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-gpbc5\" (UID: \"3ae31749-2655-4595-8089-6972597b80f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gpbc5"
Oct 30 19:35:42.387349 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:42.387316    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ae31749-2655-4595-8089-6972597b80f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-gpbc5\" (UID: \"3ae31749-2655-4595-8089-6972597b80f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gpbc5"
Oct 30 19:35:42.387514 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:42.387375    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8wph\" (UniqueName: \"kubernetes.io/projected/3ae31749-2655-4595-8089-6972597b80f0-kube-api-access-n8wph\") pod \"maas-api-revoke-sub-3e31a506-gpbc5\" (UID: \"3ae31749-2655-4595-8089-6972597b80f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gpbc5"
Oct 30 19:35:42.387514 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:42.387458    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ae31749-2655-4595-8089-6972597b80f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-gpbc5\" (UID: \"3ae31749-2655-4595-8089-6972597b80f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gpbc5"
Oct 30 19:35:42.388119 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:42.388097    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ae31749-2655-4595-8089-6972597b80f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-gpbc5\" (UID: \"3ae31749-2655-4595-8089-6972597b80f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gpbc5"
Oct 30 19:35:42.389637 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:42.389616    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ae31749-2655-4595-8089-6972597b80f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-gpbc5\" (UID: \"3ae31749-2655-4595-8089-6972597b80f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gpbc5"
Oct 30 19:35:42.443434 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:42.443409    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8wph\" (UniqueName: \"kubernetes.io/projected/3ae31749-2655-4595-8089-6972597b80f0-kube-api-access-n8wph\") pod \"maas-api-revoke-sub-3e31a506-gpbc5\" (UID: \"3ae31749-2655-4595-8089-6972597b80f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gpbc5"
Oct 30 19:35:42.485905 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:42.485881    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gpbc5"
Oct 30 19:35:42.850202 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:42.850176    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gpbc5"]
Oct 30 19:35:42.852020 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:35:42.851991    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae31749_2655_4595_8089_6972597b80f0.slice/crio-76d010686903285246ff8fa2f0bb90acfe03a65d1f02f094d3424f14149d2748 WatchSource:0}: Error finding container 76d010686903285246ff8fa2f0bb90acfe03a65d1f02f094d3424f14149d2748: Status 404 returned error can't find the container with id 76d010686903285246ff8fa2f0bb90acfe03a65d1f02f094d3424f14149d2748
Oct 30 19:35:43.028760 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:43.028719    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gpbc5" event={"ID":"3ae31749-2655-4595-8089-6972597b80f0","Type":"ContainerStarted","Data":"76d010686903285246ff8fa2f0bb90acfe03a65d1f02f094d3424f14149d2748"}
Oct 30 19:35:44.047644 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:44.047591    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gpbc5" event={"ID":"3ae31749-2655-4595-8089-6972597b80f0","Type":"ContainerStarted","Data":"306d2c2dcce41779b54017403c40511f875babec6fb27c65f0f0f9aeac4327ea"}
Oct 30 19:35:45.103328 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:45.103290    4924 generic.go:356] "Generic (PLEG): container finished" podID="3ae31749-2655-4595-8089-6972597b80f0" containerID="306d2c2dcce41779b54017403c40511f875babec6fb27c65f0f0f9aeac4327ea" exitCode=0
Oct 30 19:35:45.104046 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:45.104016    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gpbc5" event={"ID":"3ae31749-2655-4595-8089-6972597b80f0","Type":"ContainerDied","Data":"306d2c2dcce41779b54017403c40511f875babec6fb27c65f0f0f9aeac4327ea"}
Oct 30 19:35:45.710511 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:35:45.710478    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:35:46.245631 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:46.245608    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gpbc5"
Oct 30 19:35:46.385036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:46.384947    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ae31749-2655-4595-8089-6972597b80f0-kube-api-access-n8wph\" (UniqueName: \"kubernetes.io/projected/3ae31749-2655-4595-8089-6972597b80f0-kube-api-access-n8wph\") pod \"3ae31749-2655-4595-8089-6972597b80f0\" (UID: \"3ae31749-2655-4595-8089-6972597b80f0\") "
Oct 30 19:35:46.385231 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:46.385034    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ae31749-2655-4595-8089-6972597b80f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ae31749-2655-4595-8089-6972597b80f0-maas-api-xks-ca\") pod \"3ae31749-2655-4595-8089-6972597b80f0\" (UID: \"3ae31749-2655-4595-8089-6972597b80f0\") "
Oct 30 19:35:46.385231 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:46.385097    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ae31749-2655-4595-8089-6972597b80f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ae31749-2655-4595-8089-6972597b80f0-maas-api-openshift-ca\") pod \"3ae31749-2655-4595-8089-6972597b80f0\" (UID: \"3ae31749-2655-4595-8089-6972597b80f0\") "
Oct 30 19:35:46.385539 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:46.385516    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ae31749-2655-4595-8089-6972597b80f0-maas-api-openshift-ca" pod "3ae31749-2655-4595-8089-6972597b80f0" (UID: "3ae31749-2655-4595-8089-6972597b80f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:35:46.387214 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:46.387186    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ae31749-2655-4595-8089-6972597b80f0-maas-api-xks-ca" pod "3ae31749-2655-4595-8089-6972597b80f0" (UID: "3ae31749-2655-4595-8089-6972597b80f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:35:46.387316 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:46.387235    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae31749-2655-4595-8089-6972597b80f0-kube-api-access-n8wph" pod "3ae31749-2655-4595-8089-6972597b80f0" (UID: "3ae31749-2655-4595-8089-6972597b80f0"). InnerVolumeSpecName "kube-api-access-n8wph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:46.486030 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:46.486003    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8wph\" (UniqueName: \"kubernetes.io/projected/3ae31749-2655-4595-8089-6972597b80f0-kube-api-access-n8wph\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:46.486030 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:46.486027    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ae31749-2655-4595-8089-6972597b80f0-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:46.486254 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:46.486037    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ae31749-2655-4595-8089-6972597b80f0-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:47.115319 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:47.115232    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gpbc5"
Oct 30 19:35:47.115319 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:47.115273    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gpbc5" event={"ID":"3ae31749-2655-4595-8089-6972597b80f0","Type":"ContainerDied","Data":"76d010686903285246ff8fa2f0bb90acfe03a65d1f02f094d3424f14149d2748"}
Oct 30 19:35:47.115527 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:47.115360    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76d010686903285246ff8fa2f0bb90acfe03a65d1f02f094d3424f14149d2748"
Oct 30 19:35:53.102405 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:53.102298    4924 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v" podUID="026b280e-c549-4143-a61d-37e4d48f97b3" containerName="main" containerID="cri-o://f974c7ebfcbf892f37313b7668e0fedad73eaea3d98f2b412995b12d1a2827b5" gracePeriod=45
Oct 30 19:35:53.354448 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:53.354390    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"
Oct 30 19:35:53.453773 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:53.453745    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-model-cache\") pod \"026b280e-c549-4143-a61d-37e4d48f97b3\" (UID: \"026b280e-c549-4143-a61d-37e4d48f97b3\") "
Oct 30 19:35:53.453925 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:53.453862    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/026b280e-c549-4143-a61d-37e4d48f97b3-kube-api-access-n95r7\" (UniqueName: \"kubernetes.io/projected/026b280e-c549-4143-a61d-37e4d48f97b3-kube-api-access-n95r7\") pod \"026b280e-c549-4143-a61d-37e4d48f97b3\" (UID: \"026b280e-c549-4143-a61d-37e4d48f97b3\") "
Oct 30 19:35:53.453925 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:53.453906    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-tmp-dir\") pod \"026b280e-c549-4143-a61d-37e4d48f97b3\" (UID: \"026b280e-c549-4143-a61d-37e4d48f97b3\") "
Oct 30 19:35:53.454023 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:53.453976    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-home\" (UniqueName: \"kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-home\") pod \"026b280e-c549-4143-a61d-37e4d48f97b3\" (UID: \"026b280e-c549-4143-a61d-37e4d48f97b3\") "
Oct 30 19:35:53.454023 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:53.454012    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-model-cache" pod "026b280e-c549-4143-a61d-37e4d48f97b3" (UID: "026b280e-c549-4143-a61d-37e4d48f97b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:53.454163 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:53.454037    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-dshm\") pod \"026b280e-c549-4143-a61d-37e4d48f97b3\" (UID: \"026b280e-c549-4143-a61d-37e4d48f97b3\") "
Oct 30 19:35:53.454163 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:53.454106    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/026b280e-c549-4143-a61d-37e4d48f97b3-tls-certs\" (UniqueName: \"kubernetes.io/secret/026b280e-c549-4143-a61d-37e4d48f97b3-tls-certs\") pod \"026b280e-c549-4143-a61d-37e4d48f97b3\" (UID: \"026b280e-c549-4143-a61d-37e4d48f97b3\") "
Oct 30 19:35:53.454287 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:53.454190    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-tmp-dir" pod "026b280e-c549-4143-a61d-37e4d48f97b3" (UID: "026b280e-c549-4143-a61d-37e4d48f97b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:53.454287 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:53.454263    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-home" pod "026b280e-c549-4143-a61d-37e4d48f97b3" (UID: "026b280e-c549-4143-a61d-37e4d48f97b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:53.454481 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:53.454462    4924 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-model-cache\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:53.454556 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:53.454512    4924 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-tmp-dir\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:53.454556 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:53.454529    4924 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-home\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:53.456157 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:53.456133    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-dshm" pod "026b280e-c549-4143-a61d-37e4d48f97b3" (UID: "026b280e-c549-4143-a61d-37e4d48f97b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:35:53.456237 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:53.456152    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/026b280e-c549-4143-a61d-37e4d48f97b3-tls-certs" pod "026b280e-c549-4143-a61d-37e4d48f97b3" (UID: "026b280e-c549-4143-a61d-37e4d48f97b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:35:53.456316 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:53.456297    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026b280e-c549-4143-a61d-37e4d48f97b3-kube-api-access-n95r7" pod "026b280e-c549-4143-a61d-37e4d48f97b3" (UID: "026b280e-c549-4143-a61d-37e4d48f97b3"). InnerVolumeSpecName "kube-api-access-n95r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:35:53.555597 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:53.555564    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n95r7\" (UniqueName: \"kubernetes.io/projected/026b280e-c549-4143-a61d-37e4d48f97b3-kube-api-access-n95r7\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:53.555597 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:53.555593    4924 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/026b280e-c549-4143-a61d-37e4d48f97b3-dshm\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:53.555597 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:53.555603    4924 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/026b280e-c549-4143-a61d-37e4d48f97b3-tls-certs\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:35:54.150579 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:54.150541    4924 generic.go:356] "Generic (PLEG): container finished" podID="026b280e-c549-4143-a61d-37e4d48f97b3" containerID="f974c7ebfcbf892f37313b7668e0fedad73eaea3d98f2b412995b12d1a2827b5" exitCode=0
Oct 30 19:35:54.151078 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:54.150591    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"
Oct 30 19:35:54.151078 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:54.150628    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v" event={"ID":"026b280e-c549-4143-a61d-37e4d48f97b3","Type":"ContainerDied","Data":"f974c7ebfcbf892f37313b7668e0fedad73eaea3d98f2b412995b12d1a2827b5"}
Oct 30 19:35:54.151078 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:54.150664    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v" event={"ID":"026b280e-c549-4143-a61d-37e4d48f97b3","Type":"ContainerDied","Data":"854ee9085bc73e9a83bcce3bb2abd1c922fd962575e32a23370119db2bfebda1"}
Oct 30 19:35:54.151078 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:54.150681    4924 scope.go:122] "RemoveContainer" containerID="f974c7ebfcbf892f37313b7668e0fedad73eaea3d98f2b412995b12d1a2827b5"
Oct 30 19:35:54.159528 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:54.159507    4924 scope.go:122] "RemoveContainer" containerID="f974c7ebfcbf892f37313b7668e0fedad73eaea3d98f2b412995b12d1a2827b5"
Oct 30 19:35:54.159788 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:35:54.159768    4924 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f974c7ebfcbf892f37313b7668e0fedad73eaea3d98f2b412995b12d1a2827b5\": container with ID starting with f974c7ebfcbf892f37313b7668e0fedad73eaea3d98f2b412995b12d1a2827b5 not found: ID does not exist" containerID="f974c7ebfcbf892f37313b7668e0fedad73eaea3d98f2b412995b12d1a2827b5"
Oct 30 19:35:54.159844 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:54.159797    4924 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f974c7ebfcbf892f37313b7668e0fedad73eaea3d98f2b412995b12d1a2827b5"} err="failed to get container status \"f974c7ebfcbf892f37313b7668e0fedad73eaea3d98f2b412995b12d1a2827b5\": rpc error: code = NotFound desc = could not find container \"f974c7ebfcbf892f37313b7668e0fedad73eaea3d98f2b412995b12d1a2827b5\": container with ID starting with f974c7ebfcbf892f37313b7668e0fedad73eaea3d98f2b412995b12d1a2827b5 not found: ID does not exist"
Oct 30 19:35:54.309195 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:54.309156    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"]
Oct 30 19:35:54.355913 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:54.355872    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-6a305904-kserve-65d9fdcfc-b6r9v"]
Oct 30 19:35:54.832774 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:35:54.832737    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="026b280e-c549-4143-a61d-37e4d48f97b3" path="/var/lib/kubelet/pods/026b280e-c549-4143-a61d-37e4d48f97b3/volumes"
Oct 30 19:36:02.086089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:02.083273    4924 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-4kcpg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:36:02.086089 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:02.083359    4924 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-4kcpg" podUID="db7e180c-7647-4b96-be77-c7d913ae4270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:12.578483 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:12.578441    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d2a8c49-ht8t2"]
Oct 30 19:36:12.579036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:12.578870    4924 cpu_manager.go:414] "containerMap: removing container" podUID="026b280e-c549-4143-a61d-37e4d48f97b3" containerName="main"
Oct 30 19:36:12.579036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:12.578884    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026b280e-c549-4143-a61d-37e4d48f97b3" containerName="main"
Oct 30 19:36:12.579036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:12.578928    4924 cpu_manager.go:414] "containerMap: removing container" podUID="3ae31749-2655-4595-8089-6972597b80f0" containerName="revoke-keys"
Oct 30 19:36:12.579036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:12.578935    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ae31749-2655-4595-8089-6972597b80f0" containerName="revoke-keys"
Oct 30 19:36:12.579036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:12.579000    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="026b280e-c549-4143-a61d-37e4d48f97b3" containerName="main"
Oct 30 19:36:12.579036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:12.579018    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ae31749-2655-4595-8089-6972597b80f0" containerName="revoke-keys"
Oct 30 19:36:12.582169 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:12.582144    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2a8c49-ht8t2"
Oct 30 19:36:12.680379 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:12.680347    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:36:12.680379 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:12.680359    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:36:12.680610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:12.680389    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:36:12.680610 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:12.680426    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:36:12.682490 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:12.682469    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d2a8c49-ht8t2"]
Oct 30 19:36:12.682819 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:12.682798    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2dzv\" (UniqueName: \"kubernetes.io/projected/3b37ce10-204b-4b1e-810d-b1f82b7915f8-kube-api-access-b2dzv\") pod \"maas-api-revoke-sub-6d2a8c49-ht8t2\" (UID: \"3b37ce10-204b-4b1e-810d-b1f82b7915f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2a8c49-ht8t2"
Oct 30 19:36:12.682896 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:12.682855    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b37ce10-204b-4b1e-810d-b1f82b7915f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d2a8c49-ht8t2\" (UID: \"3b37ce10-204b-4b1e-810d-b1f82b7915f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2a8c49-ht8t2"
Oct 30 19:36:12.683017 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:12.683002    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b37ce10-204b-4b1e-810d-b1f82b7915f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d2a8c49-ht8t2\" (UID: \"3b37ce10-204b-4b1e-810d-b1f82b7915f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2a8c49-ht8t2"
Oct 30 19:36:12.783847 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:12.783809    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2dzv\" (UniqueName: \"kubernetes.io/projected/3b37ce10-204b-4b1e-810d-b1f82b7915f8-kube-api-access-b2dzv\") pod \"maas-api-revoke-sub-6d2a8c49-ht8t2\" (UID: \"3b37ce10-204b-4b1e-810d-b1f82b7915f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2a8c49-ht8t2"
Oct 30 19:36:12.784053 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:12.783865    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b37ce10-204b-4b1e-810d-b1f82b7915f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d2a8c49-ht8t2\" (UID: \"3b37ce10-204b-4b1e-810d-b1f82b7915f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2a8c49-ht8t2"
Oct 30 19:36:12.784053 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:12.783956    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b37ce10-204b-4b1e-810d-b1f82b7915f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d2a8c49-ht8t2\" (UID: \"3b37ce10-204b-4b1e-810d-b1f82b7915f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2a8c49-ht8t2"
Oct 30 19:36:12.784541 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:12.784512    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b37ce10-204b-4b1e-810d-b1f82b7915f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d2a8c49-ht8t2\" (UID: \"3b37ce10-204b-4b1e-810d-b1f82b7915f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2a8c49-ht8t2"
Oct 30 19:36:12.786373 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:12.786348    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b37ce10-204b-4b1e-810d-b1f82b7915f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d2a8c49-ht8t2\" (UID: \"3b37ce10-204b-4b1e-810d-b1f82b7915f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2a8c49-ht8t2"
Oct 30 19:36:12.840036 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:12.839953    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2dzv\" (UniqueName: \"kubernetes.io/projected/3b37ce10-204b-4b1e-810d-b1f82b7915f8-kube-api-access-b2dzv\") pod \"maas-api-revoke-sub-6d2a8c49-ht8t2\" (UID: \"3b37ce10-204b-4b1e-810d-b1f82b7915f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2a8c49-ht8t2"
Oct 30 19:36:12.899231 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:12.899204    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2a8c49-ht8t2"
Oct 30 19:36:13.266155 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:13.266124    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d2a8c49-ht8t2"]
Oct 30 19:36:13.268036 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:36:13.267995    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b37ce10_204b_4b1e_810d_b1f82b7915f8.slice/crio-f54584e3c5dff80cf010c4dda35a88e7cce9599b84f0c694bb49063087cf20ef WatchSource:0}: Error finding container f54584e3c5dff80cf010c4dda35a88e7cce9599b84f0c694bb49063087cf20ef: Status 404 returned error can't find the container with id f54584e3c5dff80cf010c4dda35a88e7cce9599b84f0c694bb49063087cf20ef
Oct 30 19:36:13.340398 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:13.340369    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2a8c49-ht8t2" event={"ID":"3b37ce10-204b-4b1e-810d-b1f82b7915f8","Type":"ContainerStarted","Data":"f54584e3c5dff80cf010c4dda35a88e7cce9599b84f0c694bb49063087cf20ef"}
Oct 30 19:36:14.346344 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:14.346307    4924 generic.go:356] "Generic (PLEG): container finished" podID="3b37ce10-204b-4b1e-810d-b1f82b7915f8" containerID="ca4b5329f13e637b837cc5c51cf00c420a5d46ad0c44952d55c7bb64829a64c3" exitCode=0
Oct 30 19:36:14.346742 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:14.346358    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2a8c49-ht8t2" event={"ID":"3b37ce10-204b-4b1e-810d-b1f82b7915f8","Type":"ContainerDied","Data":"ca4b5329f13e637b837cc5c51cf00c420a5d46ad0c44952d55c7bb64829a64c3"}
Oct 30 19:36:15.483765 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:15.483739    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2a8c49-ht8t2"
Oct 30 19:36:15.611381 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:15.611306    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b37ce10-204b-4b1e-810d-b1f82b7915f8-kube-api-access-b2dzv\" (UniqueName: \"kubernetes.io/projected/3b37ce10-204b-4b1e-810d-b1f82b7915f8-kube-api-access-b2dzv\") pod \"3b37ce10-204b-4b1e-810d-b1f82b7915f8\" (UID: \"3b37ce10-204b-4b1e-810d-b1f82b7915f8\") "
Oct 30 19:36:15.611381 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:15.611348    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b37ce10-204b-4b1e-810d-b1f82b7915f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b37ce10-204b-4b1e-810d-b1f82b7915f8-maas-api-openshift-ca\") pod \"3b37ce10-204b-4b1e-810d-b1f82b7915f8\" (UID: \"3b37ce10-204b-4b1e-810d-b1f82b7915f8\") "
Oct 30 19:36:15.611619 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:15.611401    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b37ce10-204b-4b1e-810d-b1f82b7915f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b37ce10-204b-4b1e-810d-b1f82b7915f8-maas-api-xks-ca\") pod \"3b37ce10-204b-4b1e-810d-b1f82b7915f8\" (UID: \"3b37ce10-204b-4b1e-810d-b1f82b7915f8\") "
Oct 30 19:36:15.611841 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:15.611810    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b37ce10-204b-4b1e-810d-b1f82b7915f8-maas-api-openshift-ca" pod "3b37ce10-204b-4b1e-810d-b1f82b7915f8" (UID: "3b37ce10-204b-4b1e-810d-b1f82b7915f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:15.613545 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:15.613510    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b37ce10-204b-4b1e-810d-b1f82b7915f8-maas-api-xks-ca" pod "3b37ce10-204b-4b1e-810d-b1f82b7915f8" (UID: "3b37ce10-204b-4b1e-810d-b1f82b7915f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:15.613645 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:15.613578    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b37ce10-204b-4b1e-810d-b1f82b7915f8-kube-api-access-b2dzv" pod "3b37ce10-204b-4b1e-810d-b1f82b7915f8" (UID: "3b37ce10-204b-4b1e-810d-b1f82b7915f8"). InnerVolumeSpecName "kube-api-access-b2dzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:15.710800 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:36:15.710759    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:36:15.712281 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:15.712261    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2dzv\" (UniqueName: \"kubernetes.io/projected/3b37ce10-204b-4b1e-810d-b1f82b7915f8-kube-api-access-b2dzv\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:15.712346 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:15.712283    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b37ce10-204b-4b1e-810d-b1f82b7915f8-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:15.712346 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:15.712295    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b37ce10-204b-4b1e-810d-b1f82b7915f8-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:16.357076 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:16.357040    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2a8c49-ht8t2"
Oct 30 19:36:16.357297 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:16.357057    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2a8c49-ht8t2" event={"ID":"3b37ce10-204b-4b1e-810d-b1f82b7915f8","Type":"ContainerDied","Data":"f54584e3c5dff80cf010c4dda35a88e7cce9599b84f0c694bb49063087cf20ef"}
Oct 30 19:36:16.357297 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:16.357137    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f54584e3c5dff80cf010c4dda35a88e7cce9599b84f0c694bb49063087cf20ef"
Oct 30 19:36:39.416831 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:39.416797    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 19:36:39.421980 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:39.421955    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jkgcq_60e0e397-bb2c-4637-9376-c5af871666cb/console-operator/2.log"
Oct 30 19:36:45.712025 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:36:45.711978    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:36:55.991156 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:55.991123    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9x5m"]
Oct 30 19:36:55.991578 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:55.991522    4924 cpu_manager.go:414] "containerMap: removing container" podUID="3b37ce10-204b-4b1e-810d-b1f82b7915f8" containerName="revoke-keys"
Oct 30 19:36:55.991578 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:55.991533    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b37ce10-204b-4b1e-810d-b1f82b7915f8" containerName="revoke-keys"
Oct 30 19:36:55.991664 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:55.991604    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b37ce10-204b-4b1e-810d-b1f82b7915f8" containerName="revoke-keys"
Oct 30 19:36:55.994987 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:55.994965    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9x5m"
Oct 30 19:36:56.092153 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:56.092125    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:36:56.092153 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:56.092139    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:36:56.092403 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:56.092271    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:36:56.092403 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:56.092341    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:36:56.094427 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:56.094405    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9x5m"]
Oct 30 19:36:56.125885 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:56.125860    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da5c69b-a65b-4aa8-9467-72a6829b2022-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-z9x5m\" (UID: \"3da5c69b-a65b-4aa8-9467-72a6829b2022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9x5m"
Oct 30 19:36:56.126010 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:56.125894    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da5c69b-a65b-4aa8-9467-72a6829b2022-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-z9x5m\" (UID: \"3da5c69b-a65b-4aa8-9467-72a6829b2022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9x5m"
Oct 30 19:36:56.126081 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:56.126024    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfl7c\" (UniqueName: \"kubernetes.io/projected/3da5c69b-a65b-4aa8-9467-72a6829b2022-kube-api-access-hfl7c\") pod \"maas-api-revoke-sub-f03629db-z9x5m\" (UID: \"3da5c69b-a65b-4aa8-9467-72a6829b2022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9x5m"
Oct 30 19:36:56.226730 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:56.226692    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da5c69b-a65b-4aa8-9467-72a6829b2022-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-z9x5m\" (UID: \"3da5c69b-a65b-4aa8-9467-72a6829b2022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9x5m"
Oct 30 19:36:56.226730 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:56.226731    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da5c69b-a65b-4aa8-9467-72a6829b2022-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-z9x5m\" (UID: \"3da5c69b-a65b-4aa8-9467-72a6829b2022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9x5m"
Oct 30 19:36:56.226996 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:56.226803    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfl7c\" (UniqueName: \"kubernetes.io/projected/3da5c69b-a65b-4aa8-9467-72a6829b2022-kube-api-access-hfl7c\") pod \"maas-api-revoke-sub-f03629db-z9x5m\" (UID: \"3da5c69b-a65b-4aa8-9467-72a6829b2022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9x5m"
Oct 30 19:36:56.227394 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:56.227365    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da5c69b-a65b-4aa8-9467-72a6829b2022-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-z9x5m\" (UID: \"3da5c69b-a65b-4aa8-9467-72a6829b2022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9x5m"
Oct 30 19:36:56.228970 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:56.228949    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da5c69b-a65b-4aa8-9467-72a6829b2022-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-z9x5m\" (UID: \"3da5c69b-a65b-4aa8-9467-72a6829b2022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9x5m"
Oct 30 19:36:56.282991 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:56.282924    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfl7c\" (UniqueName: \"kubernetes.io/projected/3da5c69b-a65b-4aa8-9467-72a6829b2022-kube-api-access-hfl7c\") pod \"maas-api-revoke-sub-f03629db-z9x5m\" (UID: \"3da5c69b-a65b-4aa8-9467-72a6829b2022\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9x5m"
Oct 30 19:36:56.307966 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:56.307945    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9x5m"
Oct 30 19:36:56.675884 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:56.675851    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9x5m"]
Oct 30 19:36:56.677484 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:36:56.677456    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da5c69b_a65b_4aa8_9467_72a6829b2022.slice/crio-b71cddf821fb4c9d902fa4b828167daabdc51caca6a654f3754cce01fd121c2c WatchSource:0}: Error finding container b71cddf821fb4c9d902fa4b828167daabdc51caca6a654f3754cce01fd121c2c: Status 404 returned error can't find the container with id b71cddf821fb4c9d902fa4b828167daabdc51caca6a654f3754cce01fd121c2c
Oct 30 19:36:57.600223 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:57.600190    4924 generic.go:356] "Generic (PLEG): container finished" podID="3da5c69b-a65b-4aa8-9467-72a6829b2022" containerID="d11a8574f42840ca151aac86f995d2487e893a05def3717d6dc401118204add4" exitCode=0
Oct 30 19:36:57.600620 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:57.600261    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9x5m" event={"ID":"3da5c69b-a65b-4aa8-9467-72a6829b2022","Type":"ContainerDied","Data":"d11a8574f42840ca151aac86f995d2487e893a05def3717d6dc401118204add4"}
Oct 30 19:36:57.600620 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:57.600308    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9x5m" event={"ID":"3da5c69b-a65b-4aa8-9467-72a6829b2022","Type":"ContainerStarted","Data":"b71cddf821fb4c9d902fa4b828167daabdc51caca6a654f3754cce01fd121c2c"}
Oct 30 19:36:58.756405 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:58.756375    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9x5m"
Oct 30 19:36:58.853723 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:58.853696    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3da5c69b-a65b-4aa8-9467-72a6829b2022-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da5c69b-a65b-4aa8-9467-72a6829b2022-maas-api-xks-ca\") pod \"3da5c69b-a65b-4aa8-9467-72a6829b2022\" (UID: \"3da5c69b-a65b-4aa8-9467-72a6829b2022\") "
Oct 30 19:36:58.853879 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:58.853766    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3da5c69b-a65b-4aa8-9467-72a6829b2022-kube-api-access-hfl7c\" (UniqueName: \"kubernetes.io/projected/3da5c69b-a65b-4aa8-9467-72a6829b2022-kube-api-access-hfl7c\") pod \"3da5c69b-a65b-4aa8-9467-72a6829b2022\" (UID: \"3da5c69b-a65b-4aa8-9467-72a6829b2022\") "
Oct 30 19:36:58.853879 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:58.853809    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3da5c69b-a65b-4aa8-9467-72a6829b2022-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da5c69b-a65b-4aa8-9467-72a6829b2022-maas-api-openshift-ca\") pod \"3da5c69b-a65b-4aa8-9467-72a6829b2022\" (UID: \"3da5c69b-a65b-4aa8-9467-72a6829b2022\") "
Oct 30 19:36:58.854235 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:58.854212    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3da5c69b-a65b-4aa8-9467-72a6829b2022-maas-api-openshift-ca" pod "3da5c69b-a65b-4aa8-9467-72a6829b2022" (UID: "3da5c69b-a65b-4aa8-9467-72a6829b2022"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:58.855697 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:58.855649    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3da5c69b-a65b-4aa8-9467-72a6829b2022-maas-api-xks-ca" pod "3da5c69b-a65b-4aa8-9467-72a6829b2022" (UID: "3da5c69b-a65b-4aa8-9467-72a6829b2022"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:58.855919 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:58.855897    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da5c69b-a65b-4aa8-9467-72a6829b2022-kube-api-access-hfl7c" pod "3da5c69b-a65b-4aa8-9467-72a6829b2022" (UID: "3da5c69b-a65b-4aa8-9467-72a6829b2022"). InnerVolumeSpecName "kube-api-access-hfl7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:58.955098 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:58.955047    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfl7c\" (UniqueName: \"kubernetes.io/projected/3da5c69b-a65b-4aa8-9467-72a6829b2022-kube-api-access-hfl7c\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:58.955098 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:58.955096    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da5c69b-a65b-4aa8-9467-72a6829b2022-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:58.955289 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:58.955107    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da5c69b-a65b-4aa8-9467-72a6829b2022-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:59.611937 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:59.611905    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9x5m"
Oct 30 19:36:59.611937 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:59.611928    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9x5m" event={"ID":"3da5c69b-a65b-4aa8-9467-72a6829b2022","Type":"ContainerDied","Data":"b71cddf821fb4c9d902fa4b828167daabdc51caca6a654f3754cce01fd121c2c"}
Oct 30 19:36:59.612195 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:36:59.611978    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b71cddf821fb4c9d902fa4b828167daabdc51caca6a654f3754cce01fd121c2c"
Oct 30 19:37:09.052176 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:09.052142    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tfrdq"]
Oct 30 19:37:09.098856 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:09.098826    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-tfrdq"]
Oct 30 19:37:10.838573 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:10.838533    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c46b0dc6-fa8d-468b-a7dc-49bdbb88670c" path="/var/lib/kubelet/pods/c46b0dc6-fa8d-468b-a7dc-49bdbb88670c/volumes"
Oct 30 19:37:15.760340 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:37:15.760293    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:37:16.487729 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:16.487699    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fxkl8"]
Oct 30 19:37:16.488137 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:16.488111    4924 cpu_manager.go:414] "containerMap: removing container" podUID="3da5c69b-a65b-4aa8-9467-72a6829b2022" containerName="revoke-keys"
Oct 30 19:37:16.488137 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:16.488138    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3da5c69b-a65b-4aa8-9467-72a6829b2022" containerName="revoke-keys"
Oct 30 19:37:16.488235 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:16.488222    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="3da5c69b-a65b-4aa8-9467-72a6829b2022" containerName="revoke-keys"
Oct 30 19:37:16.491202 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:16.491183    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fxkl8"
Oct 30 19:37:16.544955 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:16.544930    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fxkl8"]
Oct 30 19:37:16.568751 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:16.567826    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/398375b3-2947-462a-bc2c-b18df00b41b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-fxkl8\" (UID: \"398375b3-2947-462a-bc2c-b18df00b41b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fxkl8"
Oct 30 19:37:16.568751 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:16.567943    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/398375b3-2947-462a-bc2c-b18df00b41b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-fxkl8\" (UID: \"398375b3-2947-462a-bc2c-b18df00b41b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fxkl8"
Oct 30 19:37:16.568751 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:16.567976    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2n9s\" (UniqueName: \"kubernetes.io/projected/398375b3-2947-462a-bc2c-b18df00b41b2-kube-api-access-t2n9s\") pod \"maas-api-revoke-sub-0165d7d2-fxkl8\" (UID: \"398375b3-2947-462a-bc2c-b18df00b41b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fxkl8"
Oct 30 19:37:16.589315 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:16.589288    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:37:16.589315 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:16.589288    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:37:16.589493 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:16.589286    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:37:16.589493 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:16.589323    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:37:16.668569 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:16.668537    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/398375b3-2947-462a-bc2c-b18df00b41b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-fxkl8\" (UID: \"398375b3-2947-462a-bc2c-b18df00b41b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fxkl8"
Oct 30 19:37:16.668728 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:16.668590    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2n9s\" (UniqueName: \"kubernetes.io/projected/398375b3-2947-462a-bc2c-b18df00b41b2-kube-api-access-t2n9s\") pod \"maas-api-revoke-sub-0165d7d2-fxkl8\" (UID: \"398375b3-2947-462a-bc2c-b18df00b41b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fxkl8"
Oct 30 19:37:16.668728 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:16.668682    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/398375b3-2947-462a-bc2c-b18df00b41b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-fxkl8\" (UID: \"398375b3-2947-462a-bc2c-b18df00b41b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fxkl8"
Oct 30 19:37:16.669318 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:16.669294    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/398375b3-2947-462a-bc2c-b18df00b41b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-fxkl8\" (UID: \"398375b3-2947-462a-bc2c-b18df00b41b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fxkl8"
Oct 30 19:37:16.670931 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:16.670909    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/398375b3-2947-462a-bc2c-b18df00b41b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-fxkl8\" (UID: \"398375b3-2947-462a-bc2c-b18df00b41b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fxkl8"
Oct 30 19:37:16.723310 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:16.723283    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2n9s\" (UniqueName: \"kubernetes.io/projected/398375b3-2947-462a-bc2c-b18df00b41b2-kube-api-access-t2n9s\") pod \"maas-api-revoke-sub-0165d7d2-fxkl8\" (UID: \"398375b3-2947-462a-bc2c-b18df00b41b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fxkl8"
Oct 30 19:37:16.803752 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:16.803693    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fxkl8"
Oct 30 19:37:17.184658 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:17.184635    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fxkl8"]
Oct 30 19:37:17.187051 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:37:17.187019    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398375b3_2947_462a_bc2c_b18df00b41b2.slice/crio-d30825cf470fd68a42d2ffabb9869e4026a369eccb1f3bc6183737458588779e WatchSource:0}: Error finding container d30825cf470fd68a42d2ffabb9869e4026a369eccb1f3bc6183737458588779e: Status 404 returned error can't find the container with id d30825cf470fd68a42d2ffabb9869e4026a369eccb1f3bc6183737458588779e
Oct 30 19:37:17.719101 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:17.719053    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fxkl8" event={"ID":"398375b3-2947-462a-bc2c-b18df00b41b2","Type":"ContainerStarted","Data":"663cf3d270530d889e1529f1b9fa956e28b70dad82a9772cc5dd8c3911681dd6"}
Oct 30 19:37:17.719263 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:17.719124    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fxkl8" event={"ID":"398375b3-2947-462a-bc2c-b18df00b41b2","Type":"ContainerStarted","Data":"d30825cf470fd68a42d2ffabb9869e4026a369eccb1f3bc6183737458588779e"}
Oct 30 19:37:17.830304 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:17.830204    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fxkl8" podStartSLOduration=1.830188624 podStartE2EDuration="1.830188624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:17.828217385 +0000 UTC m=+3639.105267830" watchObservedRunningTime="2026-10-30 19:37:17.830188624 +0000 UTC m=+3639.107239070"
Oct 30 19:37:18.725164 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:18.725128    4924 generic.go:356] "Generic (PLEG): container finished" podID="398375b3-2947-462a-bc2c-b18df00b41b2" containerID="663cf3d270530d889e1529f1b9fa956e28b70dad82a9772cc5dd8c3911681dd6" exitCode=0
Oct 30 19:37:18.725376 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:18.725173    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fxkl8" event={"ID":"398375b3-2947-462a-bc2c-b18df00b41b2","Type":"ContainerDied","Data":"663cf3d270530d889e1529f1b9fa956e28b70dad82a9772cc5dd8c3911681dd6"}
Oct 30 19:37:19.864264 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:19.864240    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fxkl8"
Oct 30 19:37:19.900152 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:19.900124    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/398375b3-2947-462a-bc2c-b18df00b41b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/398375b3-2947-462a-bc2c-b18df00b41b2-maas-api-xks-ca\") pod \"398375b3-2947-462a-bc2c-b18df00b41b2\" (UID: \"398375b3-2947-462a-bc2c-b18df00b41b2\") "
Oct 30 19:37:19.900277 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:19.900161    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/398375b3-2947-462a-bc2c-b18df00b41b2-kube-api-access-t2n9s\" (UniqueName: \"kubernetes.io/projected/398375b3-2947-462a-bc2c-b18df00b41b2-kube-api-access-t2n9s\") pod \"398375b3-2947-462a-bc2c-b18df00b41b2\" (UID: \"398375b3-2947-462a-bc2c-b18df00b41b2\") "
Oct 30 19:37:19.900277 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:19.900263    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/398375b3-2947-462a-bc2c-b18df00b41b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/398375b3-2947-462a-bc2c-b18df00b41b2-maas-api-openshift-ca\") pod \"398375b3-2947-462a-bc2c-b18df00b41b2\" (UID: \"398375b3-2947-462a-bc2c-b18df00b41b2\") "
Oct 30 19:37:19.900671 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:19.900642    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/398375b3-2947-462a-bc2c-b18df00b41b2-maas-api-openshift-ca" pod "398375b3-2947-462a-bc2c-b18df00b41b2" (UID: "398375b3-2947-462a-bc2c-b18df00b41b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:19.901086 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:19.901043    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/398375b3-2947-462a-bc2c-b18df00b41b2-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:19.902360 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:19.902338    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398375b3-2947-462a-bc2c-b18df00b41b2-kube-api-access-t2n9s" pod "398375b3-2947-462a-bc2c-b18df00b41b2" (UID: "398375b3-2947-462a-bc2c-b18df00b41b2"). InnerVolumeSpecName "kube-api-access-t2n9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:19.902570 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:19.902547    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/398375b3-2947-462a-bc2c-b18df00b41b2-maas-api-xks-ca" pod "398375b3-2947-462a-bc2c-b18df00b41b2" (UID: "398375b3-2947-462a-bc2c-b18df00b41b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:20.002789 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:20.002719    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/398375b3-2947-462a-bc2c-b18df00b41b2-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:20.002789 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:20.002757    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2n9s\" (UniqueName: \"kubernetes.io/projected/398375b3-2947-462a-bc2c-b18df00b41b2-kube-api-access-t2n9s\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:20.739375 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:20.739341    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fxkl8" event={"ID":"398375b3-2947-462a-bc2c-b18df00b41b2","Type":"ContainerDied","Data":"d30825cf470fd68a42d2ffabb9869e4026a369eccb1f3bc6183737458588779e"}
Oct 30 19:37:20.739597 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:20.739395    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d30825cf470fd68a42d2ffabb9869e4026a369eccb1f3bc6183737458588779e"
Oct 30 19:37:20.739597 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:20.739349    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fxkl8"
Oct 30 19:37:27.908012 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:27.907976    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h8v98"]
Oct 30 19:37:27.908430 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:27.908382    4924 cpu_manager.go:414] "containerMap: removing container" podUID="398375b3-2947-462a-bc2c-b18df00b41b2" containerName="revoke-keys"
Oct 30 19:37:27.908430 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:27.908394    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="398375b3-2947-462a-bc2c-b18df00b41b2" containerName="revoke-keys"
Oct 30 19:37:27.908508 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:27.908482    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="398375b3-2947-462a-bc2c-b18df00b41b2" containerName="revoke-keys"
Oct 30 19:37:27.911790 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:27.911766    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h8v98"
Oct 30 19:37:27.974485 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:27.974449    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4f5f85-7e9c-4074-86ae-d6606a2870d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-h8v98\" (UID: \"4c4f5f85-7e9c-4074-86ae-d6606a2870d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h8v98"
Oct 30 19:37:27.974648 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:27.974491    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvrw4\" (UniqueName: \"kubernetes.io/projected/4c4f5f85-7e9c-4074-86ae-d6606a2870d9-kube-api-access-pvrw4\") pod \"maas-api-revoke-sub-6b19338f-h8v98\" (UID: \"4c4f5f85-7e9c-4074-86ae-d6606a2870d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h8v98"
Oct 30 19:37:27.974706 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:27.974674    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4f5f85-7e9c-4074-86ae-d6606a2870d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-h8v98\" (UID: \"4c4f5f85-7e9c-4074-86ae-d6606a2870d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h8v98"
Oct 30 19:37:28.010465 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:28.010422    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:37:28.010465 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:28.010419    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:37:28.010465 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:28.010456    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:37:28.010695 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:28.010456    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-clcpt\""
Oct 30 19:37:28.012679 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:28.012660    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h8v98"]
Oct 30 19:37:28.075671 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:28.075639    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4f5f85-7e9c-4074-86ae-d6606a2870d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-h8v98\" (UID: \"4c4f5f85-7e9c-4074-86ae-d6606a2870d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h8v98"
Oct 30 19:37:28.075852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:28.075677    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4f5f85-7e9c-4074-86ae-d6606a2870d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-h8v98\" (UID: \"4c4f5f85-7e9c-4074-86ae-d6606a2870d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h8v98"
Oct 30 19:37:28.075852 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:28.075697    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvrw4\" (UniqueName: \"kubernetes.io/projected/4c4f5f85-7e9c-4074-86ae-d6606a2870d9-kube-api-access-pvrw4\") pod \"maas-api-revoke-sub-6b19338f-h8v98\" (UID: \"4c4f5f85-7e9c-4074-86ae-d6606a2870d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h8v98"
Oct 30 19:37:28.076423 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:28.076398    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4f5f85-7e9c-4074-86ae-d6606a2870d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-h8v98\" (UID: \"4c4f5f85-7e9c-4074-86ae-d6606a2870d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h8v98"
Oct 30 19:37:28.077990 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:28.077970    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4f5f85-7e9c-4074-86ae-d6606a2870d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-h8v98\" (UID: \"4c4f5f85-7e9c-4074-86ae-d6606a2870d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h8v98"
Oct 30 19:37:28.130260 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:28.130233    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvrw4\" (UniqueName: \"kubernetes.io/projected/4c4f5f85-7e9c-4074-86ae-d6606a2870d9-kube-api-access-pvrw4\") pod \"maas-api-revoke-sub-6b19338f-h8v98\" (UID: \"4c4f5f85-7e9c-4074-86ae-d6606a2870d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h8v98"
Oct 30 19:37:28.224899 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:28.224831    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h8v98"
Oct 30 19:37:28.591413 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:28.591387    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h8v98"]
Oct 30 19:37:28.593576 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:37:28.593544    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c4f5f85_7e9c_4074_86ae_d6606a2870d9.slice/crio-145590f1915b791f85b7f1a53daa3377a2817c0f260ab865309ba1edbb13cf18 WatchSource:0}: Error finding container 145590f1915b791f85b7f1a53daa3377a2817c0f260ab865309ba1edbb13cf18: Status 404 returned error can't find the container with id 145590f1915b791f85b7f1a53daa3377a2817c0f260ab865309ba1edbb13cf18
Oct 30 19:37:28.779499 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:28.779466    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h8v98" event={"ID":"4c4f5f85-7e9c-4074-86ae-d6606a2870d9","Type":"ContainerStarted","Data":"145590f1915b791f85b7f1a53daa3377a2817c0f260ab865309ba1edbb13cf18"}
Oct 30 19:37:29.785632 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:29.785598    4924 generic.go:356] "Generic (PLEG): container finished" podID="4c4f5f85-7e9c-4074-86ae-d6606a2870d9" containerID="f4a834b3ffddae9d6e2a702309ee0a3862052921196f4b618379531f283b7f95" exitCode=0
Oct 30 19:37:29.786027 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:29.785656    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h8v98" event={"ID":"4c4f5f85-7e9c-4074-86ae-d6606a2870d9","Type":"ContainerDied","Data":"f4a834b3ffddae9d6e2a702309ee0a3862052921196f4b618379531f283b7f95"}
Oct 30 19:37:30.925810 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:30.925784    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h8v98"
Oct 30 19:37:31.006228 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:31.006200    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c4f5f85-7e9c-4074-86ae-d6606a2870d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4f5f85-7e9c-4074-86ae-d6606a2870d9-maas-api-xks-ca\") pod \"4c4f5f85-7e9c-4074-86ae-d6606a2870d9\" (UID: \"4c4f5f85-7e9c-4074-86ae-d6606a2870d9\") "
Oct 30 19:37:31.006389 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:31.006278    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c4f5f85-7e9c-4074-86ae-d6606a2870d9-kube-api-access-pvrw4\" (UniqueName: \"kubernetes.io/projected/4c4f5f85-7e9c-4074-86ae-d6606a2870d9-kube-api-access-pvrw4\") pod \"4c4f5f85-7e9c-4074-86ae-d6606a2870d9\" (UID: \"4c4f5f85-7e9c-4074-86ae-d6606a2870d9\") "
Oct 30 19:37:31.006389 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:31.006312    4924 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c4f5f85-7e9c-4074-86ae-d6606a2870d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4f5f85-7e9c-4074-86ae-d6606a2870d9-maas-api-openshift-ca\") pod \"4c4f5f85-7e9c-4074-86ae-d6606a2870d9\" (UID: \"4c4f5f85-7e9c-4074-86ae-d6606a2870d9\") "
Oct 30 19:37:31.006698 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:31.006675    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c4f5f85-7e9c-4074-86ae-d6606a2870d9-maas-api-openshift-ca" pod "4c4f5f85-7e9c-4074-86ae-d6606a2870d9" (UID: "4c4f5f85-7e9c-4074-86ae-d6606a2870d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:31.008387 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:31.008358    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c4f5f85-7e9c-4074-86ae-d6606a2870d9-maas-api-xks-ca" pod "4c4f5f85-7e9c-4074-86ae-d6606a2870d9" (UID: "4c4f5f85-7e9c-4074-86ae-d6606a2870d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:31.008510 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:31.008419    4924 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c4f5f85-7e9c-4074-86ae-d6606a2870d9-kube-api-access-pvrw4" pod "4c4f5f85-7e9c-4074-86ae-d6606a2870d9" (UID: "4c4f5f85-7e9c-4074-86ae-d6606a2870d9"). InnerVolumeSpecName "kube-api-access-pvrw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:31.072746 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:31.072667    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lq6dq"]
Oct 30 19:37:31.085458 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:31.085420    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lq6dq"]
Oct 30 19:37:31.107897 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:31.107868    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4f5f85-7e9c-4074-86ae-d6606a2870d9-maas-api-xks-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:31.107897 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:31.107894    4924 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvrw4\" (UniqueName: \"kubernetes.io/projected/4c4f5f85-7e9c-4074-86ae-d6606a2870d9-kube-api-access-pvrw4\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:31.108053 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:31.107911    4924 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4f5f85-7e9c-4074-86ae-d6606a2870d9-maas-api-openshift-ca\") on node \"ip-10-0-131-224.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:31.796112 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:31.796057    4924 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h8v98"
Oct 30 19:37:31.796112 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:31.796086    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-h8v98" event={"ID":"4c4f5f85-7e9c-4074-86ae-d6606a2870d9","Type":"ContainerDied","Data":"145590f1915b791f85b7f1a53daa3377a2817c0f260ab865309ba1edbb13cf18"}
Oct 30 19:37:31.796355 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:31.796157    4924 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="145590f1915b791f85b7f1a53daa3377a2817c0f260ab865309ba1edbb13cf18"
Oct 30 19:37:32.833735 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:32.833704    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a96237fa-3d73-4acc-bef1-35a49c783d20" path="/var/lib/kubelet/pods/a96237fa-3d73-4acc-bef1-35a49c783d20/volumes"
Oct 30 19:37:40.318292 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:40.318264    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-575997bf46-rbd6m_ea148539-543a-45f7-9a45-9351959b8b6d/authorino/0.log"
Oct 30 19:37:45.710137 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:37:45.710100    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:37:55.217186 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:55.217155    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2_9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c/copy-manifests/0.log"
Oct 30 19:37:55.288146 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:55.288121    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2_9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c/manager/0.log"
Oct 30 19:37:57.324361 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:57.324328    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-fxkl8_398375b3-2947-462a-bc2c-b18df00b41b2/revoke-keys/0.log"
Oct 30 19:37:57.562087 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:57.562032    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-gpbc5_3ae31749-2655-4595-8089-6972597b80f0/revoke-keys/0.log"
Oct 30 19:37:57.796073 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:57.796043    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-43f207c7-8cfdh_45a51b9b-5b2e-489d-903f-ea4ace4225b4/revoke-keys/0.log"
Oct 30 19:37:58.034831 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:58.034799    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-gf6xw_2185cb6f-0366-4545-9a0a-72ea26d6cadf/revoke-keys/0.log"
Oct 30 19:37:58.270182 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:58.270154    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-h8v98_4c4f5f85-7e9c-4074-86ae-d6606a2870d9/revoke-keys/0.log"
Oct 30 19:37:58.514460 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:58.514426    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6d2a8c49-ht8t2_3b37ce10-204b-4b1e-810d-b1f82b7915f8/revoke-keys/0.log"
Oct 30 19:37:58.744596 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:37:58.744567    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-z9x5m_3da5c69b-a65b-4aa8-9467-72a6829b2022/revoke-keys/0.log"
Oct 30 19:38:00.687293 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:00.687259    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-575997bf46-rbd6m_ea148539-543a-45f7-9a45-9351959b8b6d/authorino/0.log"
Oct 30 19:38:01.728575 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:01.728546    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-lpgm6_63b4b347-6aee-4bb6-80e7-7103bf398747/kuadrant-console-plugin/0.log"
Oct 30 19:38:04.351898 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:04.351869    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58d577dc79-xhbrt_630856d6-3139-44fd-958f-8d031ffb8f9a/kube-auth-proxy/0.log"
Oct 30 19:38:04.594282 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:04.594240    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-4kcpg_db7e180c-7647-4b96-be77-c7d913ae4270/istio-proxy/0.log"
Oct 30 19:38:05.910495 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:05.910455    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-jvm6x_75c996cd-4196-4e8a-b067-9fc146104887/istio-proxy/0.log"
Oct 30 19:38:06.141952 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:06.141908    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-mh9pf_99520a11-7547-4a41-90e6-f03de8d3cccd/istio-proxy/0.log"
Oct 30 19:38:06.915397 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:06.915363    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-w2l8h_cc2557fb-1287-4606-84d9-266025e6de6a/istio-proxy/0.log"
Oct 30 19:38:07.143564 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:07.143538    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-lcgr2_589c0305-3c1e-48d9-ad0a-7897628d9b57/payload-pre-processing/0.log"
Oct 30 19:38:07.380865 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:07.380837    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-52rkw_221a2c35-e283-4aef-8289-ebecb034e652/payload-processing/0.log"
Oct 30 19:38:07.623463 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:07.623430    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b44dd54dc-hqb7p_05fe2f04-29f7-4999-87ef-b30035b34883/router/0.log"
Oct 30 19:38:08.999697 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:08.999666    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-fd8zz_3ef7cef8-578d-4ce6-8479-990c27e212cb/main/0.log"
Oct 30 19:38:09.284016 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:09.283937    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-z6dn4_18bb436d-4071-4558-9428-9dfb7daebbe4/main/0.log"
Oct 30 19:38:09.865964 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:09.865934    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs_4d49737a-ecba-4943-a8c5-419f1a1f115a/main/0.log"
Oct 30 19:38:10.143553 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:10.143473    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd_a3d792df-177d-4f7a-bba8-f6351175462e/main/0.log"
Oct 30 19:38:11.161957 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:11.161921    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-575997bf46-rbd6m_ea148539-543a-45f7-9a45-9351959b8b6d/authorino/0.log"
Oct 30 19:38:15.709615 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:38:15.709580    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:38:23.997229 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:23.997201    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2_9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c/copy-manifests/0.log"
Oct 30 19:38:24.066780 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:24.066752    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9c7b5ddd8-s8sr2_9c72e2bb-50ca-4baf-9991-7cf8c7f80f1c/manager/0.log"
Oct 30 19:38:25.815456 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:25.815423    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-fxkl8_398375b3-2947-462a-bc2c-b18df00b41b2/revoke-keys/0.log"
Oct 30 19:38:26.078340 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:26.078256    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-gpbc5_3ae31749-2655-4595-8089-6972597b80f0/revoke-keys/0.log"
Oct 30 19:38:26.306095 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:26.306058    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-43f207c7-8cfdh_45a51b9b-5b2e-489d-903f-ea4ace4225b4/revoke-keys/0.log"
Oct 30 19:38:26.542578 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:26.542548    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-gf6xw_2185cb6f-0366-4545-9a0a-72ea26d6cadf/revoke-keys/0.log"
Oct 30 19:38:26.775454 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:26.775421    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-h8v98_4c4f5f85-7e9c-4074-86ae-d6606a2870d9/revoke-keys/0.log"
Oct 30 19:38:27.012077 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:27.012046    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6d2a8c49-ht8t2_3b37ce10-204b-4b1e-810d-b1f82b7915f8/revoke-keys/0.log"
Oct 30 19:38:27.253243 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:27.253218    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-z9x5m_3da5c69b-a65b-4aa8-9467-72a6829b2022/revoke-keys/0.log"
Oct 30 19:38:28.766480 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:28.766410    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-575997bf46-rbd6m_ea148539-543a-45f7-9a45-9351959b8b6d/authorino/0.log"
Oct 30 19:38:29.813793 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:29.813764    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-lpgm6_63b4b347-6aee-4bb6-80e7-7103bf398747/kuadrant-console-plugin/0.log"
Oct 30 19:38:32.214682 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:32.214648    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58d577dc79-xhbrt_630856d6-3139-44fd-958f-8d031ffb8f9a/kube-auth-proxy/0.log"
Oct 30 19:38:32.446872 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:32.446837    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-4kcpg_db7e180c-7647-4b96-be77-c7d913ae4270/istio-proxy/0.log"
Oct 30 19:38:33.695984 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:33.695951    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-jvm6x_75c996cd-4196-4e8a-b067-9fc146104887/istio-proxy/0.log"
Oct 30 19:38:33.924612 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:33.924583    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-mh9pf_99520a11-7547-4a41-90e6-f03de8d3cccd/istio-proxy/0.log"
Oct 30 19:38:34.682494 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:34.682457    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-w2l8h_cc2557fb-1287-4606-84d9-266025e6de6a/istio-proxy/0.log"
Oct 30 19:38:34.930476 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:34.930444    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-lcgr2_589c0305-3c1e-48d9-ad0a-7897628d9b57/payload-pre-processing/0.log"
Oct 30 19:38:35.155226 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:35.155195    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-52rkw_221a2c35-e283-4aef-8289-ebecb034e652/payload-processing/0.log"
Oct 30 19:38:35.392872 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:35.392839    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b44dd54dc-hqb7p_05fe2f04-29f7-4999-87ef-b30035b34883/router/0.log"
Oct 30 19:38:36.509734 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:36.509704    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-fd8zz_3ef7cef8-578d-4ce6-8479-990c27e212cb/main/0.log"
Oct 30 19:38:36.798793 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:36.798710    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-z6dn4_18bb436d-4071-4558-9428-9dfb7daebbe4/main/0.log"
Oct 30 19:38:37.403236 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:37.403209    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-t87gs_4d49737a-ecba-4943-a8c5-419f1a1f115a/main/0.log"
Oct 30 19:38:37.696094 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:37.696002    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-r4hcd_a3d792df-177d-4f7a-bba8-f6351175462e/main/0.log"
Oct 30 19:38:42.164137 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:42.164108    4924 scope.go:122] "RemoveContainer" containerID="cdd56da3d5e25b0d8900ccd192fb053d04a7417d07d7a6c755e2f9697fe84310"
Oct 30 19:38:42.172814 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:42.172796    4924 scope.go:122] "RemoveContainer" containerID="fab137135ed32d5c0f0edd15cbd5974de1806f06553535ac3120bff993388cf6"
Oct 30 19:38:45.710524 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:38:45.710489    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:38:46.058891 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:46.058801    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43f207c7-8cfdh"]
Oct 30 19:38:46.105580 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:46.105542    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43f207c7-8cfdh"]
Oct 30 19:38:46.832885 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:38:46.832850    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45a51b9b-5b2e-489d-903f-ea4ace4225b4" path="/var/lib/kubelet/pods/45a51b9b-5b2e-489d-903f-ea4ace4225b4/volumes"
Oct 30 19:39:10.171114 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:39:10.171029    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-52rkw_221a2c35-e283-4aef-8289-ebecb034e652/payload-processing/0.log"
Oct 30 19:39:10.401360 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:39:10.401331    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-52rkw_221a2c35-e283-4aef-8289-ebecb034e652/payload-processing/0.log"
Oct 30 19:39:15.710604 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:39:15.710567    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:39:45.710896 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:39:45.710861    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:39:51.756522 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:39:51.756495    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-52rkw_221a2c35-e283-4aef-8289-ebecb034e652/payload-processing/0.log"
Oct 30 19:39:52.009889 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:39:52.009813    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-52rkw_221a2c35-e283-4aef-8289-ebecb034e652/payload-processing/0.log"
Oct 30 19:39:52.353799 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:39:52.353765    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cx52t/must-gather-z4fb7"]
Oct 30 19:39:52.354181 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:39:52.354169    4924 cpu_manager.go:414] "containerMap: removing container" podUID="4c4f5f85-7e9c-4074-86ae-d6606a2870d9" containerName="revoke-keys"
Oct 30 19:39:52.354231 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:39:52.354183    4924 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c4f5f85-7e9c-4074-86ae-d6606a2870d9" containerName="revoke-keys"
Oct 30 19:39:52.354277 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:39:52.354261    4924 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c4f5f85-7e9c-4074-86ae-d6606a2870d9" containerName="revoke-keys"
Oct 30 19:39:52.357569 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:39:52.357547    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cx52t/must-gather-z4fb7"
Oct 30 19:39:52.406809 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:39:52.406782    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cx52t\"/\"openshift-service-ca.crt\""
Oct 30 19:39:52.406953 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:39:52.406845    4924 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cx52t\"/\"kube-root-ca.crt\""
Oct 30 19:39:52.452813 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:39:52.452784    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp9gj\" (UniqueName: \"kubernetes.io/projected/5f5b5859-1786-488c-8743-7f0a52bb9623-kube-api-access-zp9gj\") pod \"must-gather-z4fb7\" (UID: \"5f5b5859-1786-488c-8743-7f0a52bb9623\") " pod="openshift-must-gather-cx52t/must-gather-z4fb7"
Oct 30 19:39:52.452972 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:39:52.452900    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5f5b5859-1786-488c-8743-7f0a52bb9623-must-gather-output\") pod \"must-gather-z4fb7\" (UID: \"5f5b5859-1786-488c-8743-7f0a52bb9623\") " pod="openshift-must-gather-cx52t/must-gather-z4fb7"
Oct 30 19:39:52.453548 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:39:52.453531    4924 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cx52t\"/\"default-dockercfg-76gpr\""
Oct 30 19:39:52.455623 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:39:52.455602    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cx52t/must-gather-z4fb7"]
Oct 30 19:39:52.554402 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:39:52.554365    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5f5b5859-1786-488c-8743-7f0a52bb9623-must-gather-output\") pod \"must-gather-z4fb7\" (UID: \"5f5b5859-1786-488c-8743-7f0a52bb9623\") " pod="openshift-must-gather-cx52t/must-gather-z4fb7"
Oct 30 19:39:52.554598 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:39:52.554457    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp9gj\" (UniqueName: \"kubernetes.io/projected/5f5b5859-1786-488c-8743-7f0a52bb9623-kube-api-access-zp9gj\") pod \"must-gather-z4fb7\" (UID: \"5f5b5859-1786-488c-8743-7f0a52bb9623\") " pod="openshift-must-gather-cx52t/must-gather-z4fb7"
Oct 30 19:39:52.554819 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:39:52.554793    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5f5b5859-1786-488c-8743-7f0a52bb9623-must-gather-output\") pod \"must-gather-z4fb7\" (UID: \"5f5b5859-1786-488c-8743-7f0a52bb9623\") " pod="openshift-must-gather-cx52t/must-gather-z4fb7"
Oct 30 19:39:52.608982 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:39:52.608921    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp9gj\" (UniqueName: \"kubernetes.io/projected/5f5b5859-1786-488c-8743-7f0a52bb9623-kube-api-access-zp9gj\") pod \"must-gather-z4fb7\" (UID: \"5f5b5859-1786-488c-8743-7f0a52bb9623\") " pod="openshift-must-gather-cx52t/must-gather-z4fb7"
Oct 30 19:39:52.671126 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:39:52.671102    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cx52t/must-gather-z4fb7"
Oct 30 19:39:53.036399 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:39:53.035951    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cx52t/must-gather-z4fb7"]
Oct 30 19:39:53.037642 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:39:53.037583    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f5b5859_1786_488c_8743_7f0a52bb9623.slice/crio-d2ed79422418df86d57b5b2c8b3dd5caf17f800a636b39c7572c9399a1691eeb WatchSource:0}: Error finding container d2ed79422418df86d57b5b2c8b3dd5caf17f800a636b39c7572c9399a1691eeb: Status 404 returned error can't find the container with id d2ed79422418df86d57b5b2c8b3dd5caf17f800a636b39c7572c9399a1691eeb
Oct 30 19:39:53.039665 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:39:53.039645    4924 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:39:53.489805 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:39:53.489772    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cx52t/must-gather-z4fb7" event={"ID":"5f5b5859-1786-488c-8743-7f0a52bb9623","Type":"ContainerStarted","Data":"d2ed79422418df86d57b5b2c8b3dd5caf17f800a636b39c7572c9399a1691eeb"}
Oct 30 19:39:54.496211 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:39:54.496122    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cx52t/must-gather-z4fb7" event={"ID":"5f5b5859-1786-488c-8743-7f0a52bb9623","Type":"ContainerStarted","Data":"b08fbf602ef6f69662e0ab59194f5ebc03dc8ec7034afd980bb9ca08c1ae4dad"}
Oct 30 19:39:54.496211 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:39:54.496169    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cx52t/must-gather-z4fb7" event={"ID":"5f5b5859-1786-488c-8743-7f0a52bb9623","Type":"ContainerStarted","Data":"f2653150facdfec6381d46481d1b9d12753db8ba0b900e54220b0e9e2878e665"}
Oct 30 19:39:54.605046 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:39:54.604983    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cx52t/must-gather-z4fb7" podStartSLOduration=1.438816361 podStartE2EDuration="2.604970322s" totalImagesPullingTime="1.166153961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:54.604416002 +0000 UTC m=+3795.881466458" watchObservedRunningTime="2026-10-30 19:39:54.604970322 +0000 UTC m=+3795.882020798"
Oct 30 19:40:03.205207 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:03.205113    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vs8lx_92f38b64-26fb-42d6-826a-94cd37e8b668/global-pull-secret-syncer/0.log"
Oct 30 19:40:03.802675 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:03.802643    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-65rz2_a20a00eb-cde2-4df4-a191-0502fcaf288d/connection-checker/0.log"
Oct 30 19:40:04.245083 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:04.243713    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-f4bz7_e438abba-e5b6-41bb-bc67-1a1d322e6534/connection-checker/0.log"
Oct 30 19:40:04.682085 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:04.681491    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-wl4tz_d8842256-47fc-4dc5-91f6-65938f410080/connection-checker/0.log"
Oct 30 19:40:05.165300 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:05.165264    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5v2r9_1e0da5dc-3a50-414f-99aa-0a8bbe303a31/konnectivity-agent/0.log"
Oct 30 19:40:07.000153 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:07.000113    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-224.us-west-2.compute.internal_2e58124d8b2f46cf7f8af1e433d1d9a6/haproxy/0.log"
Oct 30 19:40:10.114091 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:10.113678    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-575997bf46-rbd6m_ea148539-543a-45f7-9a45-9351959b8b6d/authorino/0.log"
Oct 30 19:40:11.255207 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:11.255175    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-lpgm6_63b4b347-6aee-4bb6-80e7-7103bf398747/kuadrant-console-plugin/0.log"
Oct 30 19:40:11.332088 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:11.331055    4924 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cx52t/perf-node-gather-daemonset-5wjrq"]
Oct 30 19:40:11.338091 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:11.338042    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-5wjrq"
Oct 30 19:40:11.398900 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:11.398859    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02e66894-5418-42a4-800e-42dac2b9295b-sys\") pod \"perf-node-gather-daemonset-5wjrq\" (UID: \"02e66894-5418-42a4-800e-42dac2b9295b\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-5wjrq"
Oct 30 19:40:11.399126 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:11.398991    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h56m\" (UniqueName: \"kubernetes.io/projected/02e66894-5418-42a4-800e-42dac2b9295b-kube-api-access-4h56m\") pod \"perf-node-gather-daemonset-5wjrq\" (UID: \"02e66894-5418-42a4-800e-42dac2b9295b\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-5wjrq"
Oct 30 19:40:11.399126 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:11.399084    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02e66894-5418-42a4-800e-42dac2b9295b-lib-modules\") pod \"perf-node-gather-daemonset-5wjrq\" (UID: \"02e66894-5418-42a4-800e-42dac2b9295b\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-5wjrq"
Oct 30 19:40:11.399126 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:11.399102    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/02e66894-5418-42a4-800e-42dac2b9295b-podres\") pod \"perf-node-gather-daemonset-5wjrq\" (UID: \"02e66894-5418-42a4-800e-42dac2b9295b\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-5wjrq"
Oct 30 19:40:11.399280 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:11.399144    4924 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/02e66894-5418-42a4-800e-42dac2b9295b-proc\") pod \"perf-node-gather-daemonset-5wjrq\" (UID: \"02e66894-5418-42a4-800e-42dac2b9295b\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-5wjrq"
Oct 30 19:40:11.416446 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:11.416412    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cx52t/perf-node-gather-daemonset-5wjrq"]
Oct 30 19:40:11.499819 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:11.499782    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02e66894-5418-42a4-800e-42dac2b9295b-sys\") pod \"perf-node-gather-daemonset-5wjrq\" (UID: \"02e66894-5418-42a4-800e-42dac2b9295b\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-5wjrq"
Oct 30 19:40:11.500010 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:11.499844    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h56m\" (UniqueName: \"kubernetes.io/projected/02e66894-5418-42a4-800e-42dac2b9295b-kube-api-access-4h56m\") pod \"perf-node-gather-daemonset-5wjrq\" (UID: \"02e66894-5418-42a4-800e-42dac2b9295b\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-5wjrq"
Oct 30 19:40:11.500010 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:11.499880    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02e66894-5418-42a4-800e-42dac2b9295b-lib-modules\") pod \"perf-node-gather-daemonset-5wjrq\" (UID: \"02e66894-5418-42a4-800e-42dac2b9295b\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-5wjrq"
Oct 30 19:40:11.500010 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:11.499895    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/02e66894-5418-42a4-800e-42dac2b9295b-podres\") pod \"perf-node-gather-daemonset-5wjrq\" (UID: \"02e66894-5418-42a4-800e-42dac2b9295b\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-5wjrq"
Oct 30 19:40:11.500010 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:11.499914    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02e66894-5418-42a4-800e-42dac2b9295b-sys\") pod \"perf-node-gather-daemonset-5wjrq\" (UID: \"02e66894-5418-42a4-800e-42dac2b9295b\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-5wjrq"
Oct 30 19:40:11.500010 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:11.499931    4924 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/02e66894-5418-42a4-800e-42dac2b9295b-proc\") pod \"perf-node-gather-daemonset-5wjrq\" (UID: \"02e66894-5418-42a4-800e-42dac2b9295b\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-5wjrq"
Oct 30 19:40:11.500010 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:11.499998    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02e66894-5418-42a4-800e-42dac2b9295b-lib-modules\") pod \"perf-node-gather-daemonset-5wjrq\" (UID: \"02e66894-5418-42a4-800e-42dac2b9295b\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-5wjrq"
Oct 30 19:40:11.500010 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:11.500000    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/02e66894-5418-42a4-800e-42dac2b9295b-proc\") pod \"perf-node-gather-daemonset-5wjrq\" (UID: \"02e66894-5418-42a4-800e-42dac2b9295b\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-5wjrq"
Oct 30 19:40:11.500306 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:11.500005    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/02e66894-5418-42a4-800e-42dac2b9295b-podres\") pod \"perf-node-gather-daemonset-5wjrq\" (UID: \"02e66894-5418-42a4-800e-42dac2b9295b\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-5wjrq"
Oct 30 19:40:11.559331 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:11.559256    4924 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h56m\" (UniqueName: \"kubernetes.io/projected/02e66894-5418-42a4-800e-42dac2b9295b-kube-api-access-4h56m\") pod \"perf-node-gather-daemonset-5wjrq\" (UID: \"02e66894-5418-42a4-800e-42dac2b9295b\") " pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-5wjrq"
Oct 30 19:40:11.650801 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:11.650766    4924 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-5wjrq"
Oct 30 19:40:12.050607 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:12.050579    4924 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cx52t/perf-node-gather-daemonset-5wjrq"]
Oct 30 19:40:12.053726 ip-10-0-131-224 kubenswrapper[4924]: W1030 19:40:12.053692    4924 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod02e66894_5418_42a4_800e_42dac2b9295b.slice/crio-c7628c606a0b81891dfee9a3bc9bea7bc70881fb6bc360043b8a442e22fea1fe WatchSource:0}: Error finding container c7628c606a0b81891dfee9a3bc9bea7bc70881fb6bc360043b8a442e22fea1fe: Status 404 returned error can't find the container with id c7628c606a0b81891dfee9a3bc9bea7bc70881fb6bc360043b8a442e22fea1fe
Oct 30 19:40:12.667023 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:12.666965    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-5wjrq" event={"ID":"02e66894-5418-42a4-800e-42dac2b9295b","Type":"ContainerStarted","Data":"9d52223280c04b6bec12684a4ec14b9d7bf5ce170752c899d13b3b3cabc98d02"}
Oct 30 19:40:12.667534 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:12.667044    4924 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-5wjrq" event={"ID":"02e66894-5418-42a4-800e-42dac2b9295b","Type":"ContainerStarted","Data":"c7628c606a0b81891dfee9a3bc9bea7bc70881fb6bc360043b8a442e22fea1fe"}
Oct 30 19:40:12.667534 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:12.667128    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-5wjrq"
Oct 30 19:40:12.776758 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:12.776694    4924 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-5wjrq" podStartSLOduration=1.77667938 podStartE2EDuration="1.77667938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:12.775982819 +0000 UTC m=+3814.053033276" watchObservedRunningTime="2026-10-30 19:40:12.77667938 +0000 UTC m=+3814.053729829"
Oct 30 19:40:15.710903 ip-10-0-131-224 kubenswrapper[4924]: E1030 19:40:15.710860    4924 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-xcwnk" containerName="extractor"
Oct 30 19:40:18.112706 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:18.112673    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-plkxx_80e45018-1682-47cd-9523-dbeaf35a39cd/cluster-monitoring-operator/0.log"
Oct 30 19:40:18.684654 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:18.684628    4924 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cx52t/perf-node-gather-daemonset-5wjrq"
Oct 30 19:40:20.047350 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:20.047272    4924 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf6xw"]
Oct 30 19:40:20.094669 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:20.094637    4924 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf6xw"]
Oct 30 19:40:20.833339 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:20.833306    4924 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2185cb6f-0366-4545-9a0a-72ea26d6cadf" path="/var/lib/kubelet/pods/2185cb6f-0366-4545-9a0a-72ea26d6cadf/volumes"
Oct 30 19:40:21.811631 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:21.811591    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jrwzp_93234f2e-e57d-4698-8206-a4d00a8095f1/node-exporter/0.log"
Oct 30 19:40:22.229051 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:22.229015    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jrwzp_93234f2e-e57d-4698-8206-a4d00a8095f1/kube-rbac-proxy/0.log"
Oct 30 19:40:22.638652 ip-10-0-131-224 kubenswrapper[4924]: I1030 19:40:22.638621    4924 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jrwzp_93234f2e-e57d-4698-8206-a4d00a8095f1/init-textfile/0.log"